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assessment process which celebrates diversity in student responses. 
The process depends on informed professional judgment, shared 
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the rationale and literature that informed the work, and the apparent 
impact of the project, as it evolved from September 1989 to May 1992. 
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positive effects on learning and teaching, and that further 
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ABSTRACT 



The Calgary Board of Education and the Calgary Catholic Board of 
Education have developed indicators of quality student performance for the 
assessment of school art composition and mathematical problem solving. 
Practising teachers and school system specialists in axt and mathematics have 
developed and classroom-tested materials to identify and document quality 
indicators. The resulting qualitative materials describe a performance 
assessment process which celebrates diversity in student responses. The 
process depends on informed professional judgement, shared exemplars and 
vocabulary, and student involvement in self-assessment. 

This report describes the beginning and development of the project, the 
rationale and literature that informed the work, and the apparent impact of 
the project, as the project evolved from September 1989 to May 1992. The 
project results to date indicate that performance assessment has positive 
effects on learning and teaching, and that further explorations and 
applications of the quality indicators project would be educationally 
significant. 
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CHAPTER 1: INTRODUCTION 



This chapter describes the background and purposes of the Calgary 
Educational Quality Indicators (EQI) Project, along with the assumptions, 
definitions, and methodologies that characterize the project activities. 

Background 

In the Spring of 1989, officials from the Calgary Board of Education 
(CBE), the Calgary Catholic Board of Education (CCEE) and Alberta 
Education negotiated a contract as part of the provincial EQI Initiative 
that conunitted the parties to a collaborative project entitled "The 
Development of Indicators of Academic Excellence" (called the EQI 
Project or the "project" in this report). The purpose of the project v^as to 
develop and pilot test indicators of quality student work in areas of school 
art and mathematics. Personnel v^ere identified and assigned to the project 
at the beginning of the 1989 - 90 school year. 

The Alberta art curriculum includes depiction and composition from grades 
1 through 12. Art is a compulsory course for students only in grades 1 to 
6. Art achievement is often assessed in terms of student effort, attitude, 
participation or independent teacher professional judgement. There are no 
assessment devices available that v^ere based on commonly understood 
expert standards regarding the quality of student artwork. 

The articulation and communication of quality indicators in student 
artwork is intended to result in improved teaching behaviours and 
improved student learning. The project provides documentation of those 
observable qualities in student art which should be recognized and 
encouraged and which assist in the development of teacher expertise in 
the area of art. 

Problem solving was selected for the development of quality indicators 
because of the importance of problem-solving skills in the mathematics 
program and because of a predicted transfer of the problem-solving 
practices to other subject areas. Problem solving in mathematics is both a 
specific content strand as well as a unifying construct for the entire 
program. This is evident in Alberta Education curriculum documents for 
elementary, junior high, and senior high school mathematics courses. The 
assessment of problem solving involves the examination of the thinking 
processes that underlie student answers to problems. 

Traditional assessment practice in mathematics has been most concerned 
with the product or answer given. Mathematics teachers and curriculum 
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experts encourage the assessment of problem-solving processes, but few 
materials were available to all teachers to assist in this task. 

Context 

Calgary is a city of 700,000 people located 100 kilometres east of the 
Rocky Mountains in central Alberta. There are two publicly-supported 
school systems to serve the urban area: the Calgary Board of Education, 
with an enrolment of over 94,000 students and the Calgary Catholic 
Board of Education, with an enrolment of about 30,000 students. The 
CBE ?jid the CCBE are respectively the third largest and tlie twenty-fifth 
largest school systems in the country (Canadian Education Association, 
1992). Both school systems provide subject area assistance to classroom 
teachers through central office specialists in most school subject areas. 
These specialists are involved in the ^^ganization and delivery of teacher 
inservice programs, the selection and development of materials to support 
teaching and assessment, and the recommendation of school system 
policies that affect their subject specialties. There is a long history of 
professional collaboration between the two school boards. 

Rationale 

The CBE and CCBE are committed to discovering and using the best 
possible ways to gather information about and report on student 
achievement in their schools. The EQI Project provided an opportunity to 
emphasize this commitment. 

The CBE and the CCBE report on the achievement of individual students 
and of groups of students using a variety of techniques and materials. 
Some measures and reports about student achievement, such as the 
Student Achievement Tests and Diploma Examinations, are prepared and 
distributed by Alberta Education. Other information regarding student 
achievement is prepared and distributed by school system officials, by 
school administrators and by classroom teachers. 

Most of the reports about student achievement are in the foirn of results on 
paper-and-pencil tests where there is a single correct answer. Student 
performances which are creative or original escape inclusion in traditional 
tests, although they are powerful indicators of student success. Student 
achievements in creative expression, critical thinking, and problem solving 
fall into this performance category. These complex student performances 
are difficult to report on using numbers, averages, and percentiles. 
Professional educators require new alternatives for assessing and reporting 
on student achievement that is demonstrated in complex performances. 
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This project addresses the lack of such alternatives through the 
development of quality indicators in art and mathematics. 

Teachers and fine arts specialists in the CCBE had identified a need for 
assessment materials for the visual arts. They requested help with the task 
of identifying quality in the art works that students produce. This request 
fomied the basis of one part of the EQI Project proposal. The contract 
developers were also aware of requests for assessment materials for 
problem, solving in mathematics. Problem solving has recently become a 
very important element of mathematics courses throughout the grades, and 
teachers and students needed new materials to help demonstrate student 
achievement in this content area. 

Purposes 

The purposes of the EQI Project were the following: 

• to identify iudicators of quality work which can be applied in the 
assessment of student achievement in areas of art and mathematics, 

• to identify conditions that appear to enhance or limit quality 
achievement, 

• to develop strategies for collecting quality student work which 
include recorded interviews with teachers and students, and 

• to examine quality work with emphasis on alternative assessment 
strategies to paper and pencil testing. 

Assumptions 

EQI project advisors and practising teachers appeared to have common 
understandings or assumptions about learning, teaching and assessment 
that served as a background from which to proceed with the project work. 
These three elements organize our assumptions. 

About Learning 

There is a difference between information and knowledge, although 
students attain both as a result of schooling. Information refers to the 
content of the school curriculum that can be memorized and recalled for 
presentation. Knowledge, on the other hand, is the result of a student's 
making meaning out of information and experience. The possession of 
knowledge is demonstrated in the performance of tasks that challenge 
students to apply what they know to a new situation. 
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About Teaching 



Teachers are responsible for providing students with learning experiences 
that assist students in making meaning and in practising their learned skills. 
The teacher is like a coach or mentor. 

Teachers can identify quality in student work, although they may not all 
be able to articulate the characteristics that indicate quality, or 
communicate about these characteristics in the same way. 

About Assessment 

Traditional paper-and-pencil tests which require simple, single answers to 
highly structured situations cannot adequately assess student knowledge. 
Assessment practice has an impact on teaching practice. 

Student performances include some unexpected responses. Teachers need 
to be able to analyze these responses, place them in context, and comment 
appropriately. Practising expert teachers can recognize qualitative 
differences among student performances. 

Teachers and students can develop skills in "connoisseurship" regarding 
the performance of complex tasks. That is, they can develop "a highly 
differentiated array of anticipatory schema that enable one to discern 
qualities and relationships that others ... are less likely to see" (Eisner, 1985, 
p. 153). Samples of excellent performances can be used effectively to help 
develop connoisseurship skills. 

Learners develop appropriate self-assessment skills with coaching and 
practice. 

Definitions 

In this project quality indicators are defined as observable 
characteristics of excellence established through consensus of 
professional judgement among practising teachers. 

Assessment is gathering information or observable evidence of what a 
learner can do. Evaluation is making judgements based on interpreting 
assessment information. Assessment and evaluation are part of the 
teaching process. 
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Design 



This project was designed as action research: we began with the 
experience of actual practice, and reviewed this experience to develop 
frameworks and proposals for further study. In this section, we briefly 
describe activities, structure and evaluation as these were shown in 
project design. 

Activities 

The project design incorporated three phases: development, field testing, 
and pilot testing, which correspond to the three years of project funding. 
Activities in the developmental phase included the review of the literature, 
establishment of committees, and preparation of initial structures for 
indicators of quality student achievement. Committees of teachers and 
subject specialists reviewed student work, identified characteristics of 
quality work, considered alternative approaches for recording assessment 
information, and proposed audv^ities and materials for use in classrooms. 

In the field-testing phase, project personnel tried a variety of materials and 
approaches in classrooms. Promising approaches were pursued and 
evidence of the results of field tests was collected. This evidence was used 
to revise the materials and to structure plans for pilot testing. 

Pilot tests were carried out by teachers who had no previous experience 
with the EQI project. The pilot teachers tried out the assessment materials 
during their regular teaching practice and during specifically designed 
classroom activities. Project advisors provided inservice prior to piloting 
and support to pilot teachers during the pilot period. 

Structure 

Art and mathematics subject specialists from the two school systems 
provided r 'vice, support and direction for the project. These specialists 
were the advisors to the project as well as participants in project activities. 

Classroom teachers, identified by subject specialists for their expert 
classroom practice and professional commitment, provided the 
developmental work, the testing of materials in classrooms, and thoughtful 
reflections and evaluations. School principals approved each teacher s 
participation and then the teachers volunteered to participate. The first 
team of teachers was asked to work with the project for the first two years. 
Because much of their work was action research into their own practice, 
they are called "teacher researchers." Teachers whose primary role was to 
try out specific materials and practices in their classrooms are called "field 
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test teachers" and ''pilot test teachers." For every project task, the project 
teachers represented both the selected grade levels of the study and the 
two school jurisdictions. 

Senior superintendents and education officials served on project steering 
committees that provided coordination between the two school 
jurisdictions and Alberta Education, and provided direction to the project 
manager. An advisory committee provided a communications link to other 
educational stakeholders. Members of the advisory committee included: 
university professors in art and in mathematics, leaders in the art and 
mathematics communities, parent representatives, teachers' association 
representatives, and members of the business community. 

A local advisory committee — consisting of representatives from the 
University of Calgary, the Alberta Teachers' Association, parent councils in 
both school boards, the arts and mathematics communities, and the 
business community in Calgary — assisted in providing advice and 
feedback regularly throughout the project. The project personnel who 
formed the development, advisory and steering committees for the project 
are listed in Appendix A. 

Evaluation 

Project evaluation was both formative and summative throughout the 
project. The project evaluation plan was based on a "Context, Inputs, 
Process and Product" (CIPP) framework (Stufflebeam & Webster, 1988). 
This framework structures evaluation questions around the four issues: the 
contextual environment for the project, the input resources that were 
required and/or used in the project, the processes and activities that shaped 
the project, and finally, the products of the project and their perceived 
usefulness and value. Teachers and advisors regularly considered 
questions about the processes and products of the project. Their 
recommendations were used to make changes to activities and materials 
throughout the project. 

Both quantitative and qualitative data were gathered about the effects of 
the project in classrooms. Many different information-gathering methods 
were used: students were videotaped and audio taped during problem- 
solving and ai t-making sessions; written responses were requested; survey 
data were collected; and an external interviewer spoke with advisors. 
Quantitative data were collected regarding: teacher behaviours and beliefs 
about art and problem solving; student attitudes, dispositions and scores 
on problem solving; and teacher reports of practice and the impact of 
EQI-Math on their teaching. Qualitative data were collected regarding: 
teacher beliefs and practices in art and mathematics, before and after 
piloting; student responses to EQI materials; teacher and student journals 
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in art; anecdotal records of student problem solving; descriptions of the 
mathematics classroom environments; changes reported by teachers as a 
result of using EQI materials; and impressions of the project formed by 
project advisors. 

An external consulting agency completed a formal evaluation of project 
utility, management, and impact. This evaluation is found in Appendix D. 

In this report, data from more than one source have been used to develop 
conclusions and recommendations. The accumulated data have not all 
been analyzed at this time. However, original data have been maintained 
in project archives for future reference. 

Scope 

The art component of the project (EQI-Art) focused on the depiction and 
composition components of the Albena Education art curriculum insofar as 
these are displayed in completed student art works. Student art works 
from grades 2 and 5 are the elementary exemplars. At the junior and senior 
high school levels, art works reflect all grade levels, because many schools 
offer only one art course or offer courses for students from any of the three 
grades in that division. 

EQI-Math focused on problem solving, which is both a content area and a 
method of teaching in the Alberta Education mathematics curricula. Grades 
2, 5, 8, and 10 were selected to provide a representative sample of student 
achievement for the EQI Project. Grades 2, 5, and 8 are midway through 
each of the school levels — primary, late elementary, and junior high. 
Grade 10 was identified in place of grade 1 1 because there was a new 
curriculum implementation affecting grade 1 1 mathematics courses during 
the EQI Project years. 

Limitations 

School system specialists identified successful, professionally involved 
practising teachers for the EQI Project. School principals approved each 
teacher's participation and then the teachers volunteered to participate. 

Twenty-six practising teachers from the CBE and the CCBE were teacher 
researchers during the initial phase of the project. Another 24 teachers 
field tested or pilot tested project materials in their classrooms. These 50 
teachers represent a very small percentage of the more than 5,000 
classroom teachers in the two school systems. 
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Teachers piloted the project materials in 16 schools during the first half of 
the 1991-92 school year. Two classes at each grade level pilot tested the 
EQI-Art and EQI-Math materials. This is a very small pilot test, limited by 
the resources available for the project. The project teachers were selected 
because of their expertise and commitment to professional development. 
This fact may limit the transfer of experiences from these teachers to the 
population of teachers in general. 

Overview 

This chapter has described the background to the Calgary EQI Project, its 
purposes, the organization and scope of the project. Chapter 2 includes a 
discussion of the related literature and concludes with the implications of 
the literature for the structure and activities of the EQI Project, Chapters 3 
and 4 discuss the activities and findings from the EQI- Art and the 
EQI-Math projects respectively. Chapter 5 brings the two projects 
together with a discussion of the findings of the project as a whole and 
recommendations for future activities. 
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CHAPTER 2: RELATED LITERATURE 



The teachers and advisors working on the EQI Project began their work 
based on their personal practical knowledge of schools and schooling. 
The project coordinator responded to and supported the committee 
deliberations by compiling hterature of interest to the committees in the 
areas of learning, teaching and assessment. The following discussion 
incorporates the key aspects of that literature, and concludes with 
implications for the EQI Project. 

Learning 

The EQI Project developed new materials for the assessment of student 
learning in areas of art and mathematics. The understanding about 
learning is an important basis for the project's work. This section addresses 
this understanding in two subsections: 1) from the perspective of goals of 
education and schooling, and 2) as part of the view of learning in 
cognitive theory. 

Goals of Education and Schooling 

The aim of education in Alberta is to develop the knowledge, the skills, and 
the positive attitudes of individuals, so that tiiey will be self-confident, 
capable and committed to setting goals, making informed choices and 
acting in ways that will improve their own lives and the life of their 
community (Alberta Education, 1991). 

The learner is expected not only to have knowledge but also be able to 
use knowledge in significant ways. "What matters is not tiiat one can read, 
but rather that one does read and that what one reads is worth reading" 
(Eisner, 1983, p. 49). Alberta's goals of secondary education specify tiiat 
schools will assist students to accumulate a wide variety of knowledge and 
skills, and also intend that students: 

• develop the ability to think conceptually, critically and creatively; to 
acquire and apply problem-solving skills; to apply principles of logic; 
and to use different modes of inquiry; 

• assume increasing responsibility for independent and continuous 
learning, and develop positive attitudes toward learning while in 
school, in preparation for self-directed, lifelong educational 
experiences (Alberta Education, 1991, p. 4). 
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Educators and others often use the term "literacy" in detming the goals of 
education. Discussions about literacy reiterate the message that the 
possession of "basic" skills is insufficient. Resnick dud Klopfer (1 989) 
extend hteracy into "competent thinking." For the Calgary Board of 
Education (1989), literacy is a necessary condition for personal growth, 
including the abilities people have to understand and to participate and 
communicate in the communities in which they find themselves. Greene 
(1983, 1990) speaks of literacy as an abihty to extend personal power over 
symbols and knowledge in a whole range of intellectual environments, the 
arts and technologies, literature, politics, and social relationships. 

Definitions of mathematical literacy imply a similar set of objectives, 
extending from the development of technical mathematical proficiency to 
the ability to understand and appreciate the contributions of mathematical 
concepts to everyday decision making, to culture and to civilization 
(National Council of Teachers of Mathematics, 1989). 

Cognitive Theory 

Currently, learner's thinking skills have become a focus of research study 
in education. Modem cognitive theory asserts that "people are not 
recorders of information but builders of knowledge structures" (Resnick & 
Klopfer, 1989, p. 4). Learning is a complex, meaning-making activity that 
can have many representations, depending on the skills, experience and 
environment of the learner. Both art making and mathematical problem 
solving are depicted as this kind of learning. Visual art is the observable 
result of cognitive decision making, and its production combines 
perception, use of visual symbol systems, technical skill, cultural 
appreciation, and emotion (MacLeod, 1989). Essays on the processes of 
problem solving (Charles & Lester, 1987; National Council of Teachers of 
Mathematics, 1989; Schoenfeld, 1989) emphasize the complexity of the 
process, including logical cognitive processes as well as collaboration with 
others, dispositional factors, and other environmental interactions in the 
depiction of problem-solving skills and processes. 

Cognitive theorists and educators endeavour to understand what is 
happening in the mind of the learner. Norris (1989) suggests that teachers 
need to take the extra time to go beyond the student answer and to 
encourage critical thinking. He points out the usefulness of talking to test- 
takers, encouraging them to reflect and reason about their answers. 
Gardner (1991) encourages the development of expert understanding in 
learners, rather than settling for "correct-answer compromises" which may 
cover up a lack of significant understanding. Skemp (1989) considers the 
difference between teaching for "instrumental understanding" which 
expects that students have formulas or habits of thought, and teaching for 
"relational understanding," which requires that students develop 
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conceptual structures to guide thinking about mathematical concepts* He 
calls on teachers to first understand concepts children hold, and then to 
help them build new, more appropriate schema. 

Metacognition, or how students evaluate and control their own thinking is 
expected to be systematically developed in schools (Romberg & 
Carpenter, 1986; Schoenfeld, 1989). Snow (1989) provides a model for 
research on assessment that includes these metacognitive processes and 
recommends assessment models that capture higher order cognitive skills 
and conative structures. Processes of metacognition engage the student in 
self-assessment and self-regulation. Schoenfeld (1989) describes this 
engagement as part of that understanding that leads to mastery. The major 
goal in mathematics should be teaching "power" or higher order thinking 
skills. He encourages the use of coaching techniques when working with 
students on problem solving in the belief that students can be taught to 
regulate their own thinking through metacognition. 

Teaching 

The teacher is the critical factor according to much of this research, which 
calls for the teacher to arrange learning environments and activities which 
"stimulate and nourish students' own mental elaborations of knowledge 
and to help them grow in their capacity to monitor and guide their own 
learning and thinking" (Resnick & Klopfer, 1989, p. 5). In this view of 
learning, teacher expertise becomes increasingly critical to student 
achievement. This section addresses how the conceptions of teaching 
have changed to value the teacher as reflective practitioner and as 
mentor and coach. 

Reflective Practitioner 

As expectations of education are sorted out in the political process, and as 
research provides information about relationships within the education 
process, paradigms or persistent pictures emerge of the educational reality 
that change only gradually over time. Shavelson (1988) and Brophy and 
Good (1986) document three paradigms that have been used to 
understand the teaching process: the "trait conception" of good teaching 
which connected performance with the appearance of personal 
characteristics in the teacher in the 1940s and 1950s; the "skills 
conception" of good teaching which focused on the microteaching skills 
and a repertoire of competencies for instruction in the 1960s and 1970s; 
and the "reflective practitioner" conception of teacher that places the 
teacher as mediator in the process of student construction cf knowledge in 
the 1980s and now into the 1990s. 
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Clark and Peterson (1986) describe the emergence of the view of the 
teacher as reflective practitioner to replace the notion of teacher as skilful 
technician. This view is consistent with the teacher as clinician and action 
researcher, which is evident in the work of Duckworth (1986), Goodlad 
(1986, 1987), Clandinin (1986), Calfee and Hiebert (1988), Costa (1989), 
Shavelson and Stem (1981), Sirotnik and Clark (1988). Teacher thinking, 
as conceptualized in this body of research, is a central set of moderating 
contextual factors that influence curriculum effectiveness, school 
effectiveness, and student achievement. Teacher thinking must be the 
focus of teacher development practices at all levels in education. 

If we wish to develop intelligent behaviour as a significant 
outcome of education, instructional strategies purposefully 
intended to develop children's metacognitive. abilities must be 
infused into our teaching methods, staff development and 
supervisory processes (Costa, 1984, p. 58). 

Mentor and Coach 

Duckworth (1986) encourages teachers to put students in direct contact 
with phenomena to be studied and to understand the sense students are 
making. She suggests that mentoring and coaching students, as a partner 
in their learning, has important, very desirable results. First, students gain 
clarity of thought by explaining themselves to others. Second, they 
determine for themselves what they want to learn. Third, students come to 
depend on themselves, they become the judges of what they know and 
believe. Fourth, students recognize the powerful experience of having 
their ideas taken seriously. Fifth, students learn an enormous amount from 
one another. In this process, learners come to recognize knowledge as a 
human product, and to be self-confident about learning. 

Assessment 

Assessment is the process of gathering information or observable evidence 
of what the learner can do. This section examines functions, need for 
change, some alternative approaches, and a systems approach related 
to assessment. 

Functions 

The root word of assessment means to "sit with" a learner, paying close 
attention to the evidence of a learner's thoughtful understanding. "And 
thoughtful understanding implies being able to do something effective, 
transformative, or novel with a problem ornovel situation'* (Wiggins, 1989, 
p, 34). The background from which a teacher can make appropriate 
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assessments is rooted in that teacher s deep understanding and abilities to 
communicate that understanding to students. Eisner (1985, 1991) defines 
that deep understanding as connoisseurship — the ability to perceive 
subtle qualities and nuances not available to most. The connoisseur must 
then use criticism, which is" ...[the] art of saying useful things about 
complex and subtle objects and events so that others less sophisticated or 
sophisticated in different ways, can see and understand what they did not 
see and understand before" (1991, p. 3). 

Resnick and Resnick (1985) discuss assessment and curriculum as the two 
key components that can be changed to effect school improvements. The 
evidence and connmon sense tell us that what is assessed will be taught. 
Although the practice of teaching to the test may be frowned on in 
educational circles, as Nickerson (1989) says, "frowning on a practice has 
seldom been an effective deterrent if the incentives to engage in it are 
strong" (p. 3). The incentives to increase test scores are substantial, 
propelled by competition for resources and increased scrutiny and criticism 
of tiie educational enterprise. 

Need for Change 

One key issue in discussions about assessment is encapsulated in the 
statement: "American children are the most tested in the world and the 
least examined" (Resnick & Resnick, 1985, p. 17). Their intention is not to 
eliminate testing, but rather to improve tests so that they demonstrate real 
achievement, what we want students to know as a result of schooling. 
Assessment determines what teachers teach, states Wiggins (1989), so 
instead of trying to fight that tendency, educators should concentrate on 
developing effective tests that examine those competencies they think are 
essential. 

If the goals of education are broad, then the tests used to assess 
achievement must address this breadth (Hargreaves, 1990; California 
Assessment Program, 1989). A reliance on only one form of assessment, 
such as the multiple choice test, cannot do the job fully. There is an 
extensive body of expressed concern over the exclusive use of multiple 
choice testing, most often highlighting the problems of teaching to the test, 
ignoring untested curricula, segmenting knowledge, "decontextualizing" 
knowledge, and downplaying the role of the teacher as professional. 
(Archbald & Newmann, 1988; Mumane & Raizen, 1988; Resnick & 
Resnick, 1985; Wiggins, 1989). 

Teaching includes the diagnosis of learning concerns and the development 
and implementation of activities to address these. As part of the teaching 
process, assessment practices must provide information to help with 
diagnosis and development activities. Assessments must contain the 
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information, and teachers must understand and use the information to 
make decisions. In the assessment of self-regulatory and motivational 
activities of learners, Snow (1989) states that some teachers know how to 
assess learner outcomes in these areas informally, and research faces the 
considerable task of refining tools for use by teachers and others to 
formalize and improve this measurement. Assessment instruments need 
careful design, need to be developed by professionals in education, and 
assistance with interpretation must be available to those who use the 
information. 

Some Alternative Approaches 

Performance assessment alternatives are suggested as a means to assess 
higher order thinking skills and to broaden assessment processes. These 
alternatives are encouraged in many subject areas and implemented 
through such vehicles as portfolios, extended performances and displays. 

One example of performance assessment is the Arts PROPEL project, a 
long-term developmental project that involves the Educational Testing 
Service, Pittsburgh Public Schools and the Rockefeller Foundation 
(Gardner, 1989; Wolf, 1989b). The PROPEL project supports the 
development of performance assessment instruments to document artistic 
learning from late elementary to high school in music, art and expressive 
writing. Some of the main tenets include: teachers having deep 
know':^dge in the subject areas; learning centred around meaningful, 
significant projects which may involve students for a significant period of 
time; assessment as crucial, but respectful of the inteUigence being 
assessed. The term PROPEL is an acronym for the three competencies to 
be assessed: production, perception, and reflection. Other examples of 
performance assessment in the arts recommend portfolio assessment for a 
variety of functions to foster artistic critiques, aesthetic reasoning, and to 
recognize the personal values espoused ir artistic expression (Gardner & 
Grunbaum, 1986). Wolf (1989a) recommends portfoUo assessments to 
promote characteristics such as increased student responsibility, an 
enlarged view of what is learned, recognition of personal progress, and the 
emergence of a developmental point of view. 

Systems Approach 

Education is a system of interactions, rather than a linear process of cause- 
effect relationships. Instead of controlling for the influence of teaching to 
the test, or practice effects, Frederiksen and Collins (1989) encourage 
educators to take advantage of "systemic validity," the direct and expected 
connection between instruction and assessment. "A systemically valid test 
is one that induces in the educational system curricular and instructional 
changes that foster the development of the cognitive skills the test is 
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designed to measure" (p. 27). A test would be systemically valid only if it 
caused improvement in those skills it was designed to measure after it was 
in place for a certain period of time. 

Frederiksen and Collins (1989) identify the directness of cognitive 
assessment and the degree of subjectivity or judgement required as 
important characteristics for systemically valid tests (p. 28). Direct 
assessment means that the cognitive skill that is of interest is directly 
evaluated as it is expressed in the performance of some extended task. This 
"directness" is the underlying tenet of performance assessment (Stiggins, 
1987, 1988), authentic assessment (Wiggins, 1989; Calfee & Hiebert, 1988; 
California Assessment Program, 1990), and portfolio usage (California 
Assessment Program, 1990; Wolf. 1989a, 1989b). 

The degree of subjectivity refers to the degree to which judgement, 
analysis, and reflection is used in assigning a score to a test performance. 
Frederiksen and Collins (1989) recommend subjective assessment for the 
development of materials to assess student performances, and the 
implementation of collaborative systems for marking and moderating 
student work. They say: 

Although it is worthwhile to develop objective tests of important cognitive 
outcomes, in general the state of the art does not permit objective tests for 
directly measuring higher order thiiiking skills, problem-solving strategies, 
and meta-cognitive abilities such as those involved in teaching, writing, 
and 'doing' mathematics (p. 29). 

The training materials for subjective assessment help to communicate to 
teachers and students the critical traits to look for in examining their own 
or other's work. This communication function is a central component of 
the testing procedures of the Assessment of Performance Unit in Britain 
(Joint Matriculation Board, 1984), and the California Assessment Program 
(CAP, 1990) and the Benchmarks program in Toronto (Toronto Board of 
Education, 1989). 

Components of a systemically valid testing system would include a set of 
tasks, a specification of primary traits of both process and product that the 
assessors know are leamable, a library of exemplars, standards, and a 
method for fostering improvement on the test, including such activities as 
student practice (Frederiksen & Collins, 1989). This assessment system fits 
the practical descriptions of the testing programs noted in Britain, 
California, and Toronto, and is a useful format for the development of 
alternative assessment techniques in the EQI project. 
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Implications for EQI Calgary 



The Calgary EQI Project addresses the cognitive outcomes in the problem- 
solving strand of mathematics and in the depiction and composition 
objectives of art. These two sets of outcomes represent particularly 
difficult challenges for assessment because of the diverse nature of the 
student outcomes and the expectation that the professional judgement of 
educators will be central to completing the assessment. The literature 
describes the need for changes to accommodate what we know about 
learning, teaching and assessment, and provides us with some promising 
new approaches for developing new assessment practices. 

The need for tests that do a better job of assessing thinking skills is 
widely recognized. The making of visual art and the solving of problems 
in mathematics both require complex thinking skills. Strategic flexibility, 
adaptive control, and achievement motivation are demonstrated in problem 
solving (Snow, 1989; National Council of Teachers of Mathematics, 1989; 
Schoenfeld, 1988; Mumane, 1988). A similar set of cognitive processes is 
suggested in the literature regarding the production of visual art (Glenn, 
1989; Wolf, 1989a). Research reviews (Norris, 1989; Siegler, 1989; 
Nickerson, 1989) suggest positive directions for improvements to testing 
methods used currently and for the development of new instruments and 
strategies to apply what is now being discovered about cognitive 
operations. 

Promising approaches that link the assessment process with instructional 
processes and focus on the development of teacher skills and judgement 
are under way in many formats around the world. Some programs of this 
nature are formal programs instituted by governing agencies and some are 
simply individual teaclier assessment activities. Psychometric researchers 
(Snow, 1989; Nickerson, 1989) and educators involved directly in school 
change efforts (Wiggins, 1989; Stiggins, 1988; Gardner, 1989; National 
Council of Teachers of Mathematics, 1989) present a consistent emphasis 
on the development of teacher expertise in the process of improving 
assessment, both in external assessment developed by governing agencies 
and in teacher developed testing. 

A "systems approach" to assessment such as described by Frederiksen and 
Collins (1989) provides a model to accommodate the demands of 
assessment in the production of visual art and in mathematical problem 
solving. First, the systems model adopts a view of learning as a complex 
meaning*making activity that has many representations, depending on 
the skills, experience and environment of the learner. Second, assessment 
is facihtated by a teacher whose informed professional judgement is 
essential for the assessment of student cognition. Third, assessment is 
focused on the actual behaviours that demonstrate the accomplishment 
of the leaming objective intended, often the extended response to a 
complex challenge or problem. 
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CHAPTER 3: EQI-ART 
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Images are records of created meaning ... the image is an attempt 
of the mind to penetrate the meaning of the world and the 
meaning of being human within it. Even in the most primitive 
attempts at the symbolic language of images, there is evidence of a 
mind at work: selecting, rejecting, arranging, creating: inventing 
the clothing for the idea. 

MacLeod (1989, p. 7) 
Introduction to EQI-Art 

Student outcomes in response to the depiction and composition objectives 
of the Alberta Education art curriculum are observed in the art produced 
by students. However, there are currently no assessment devices available 
that are based on commonly understood expert standards regarding the 
quality of student artwork. The EQI-Art project was designed to fill this 
void, by developing quality indicators that assist both teachers and 
students in their respective tasks of teaching and learning. 

The EQI project has defined quality indicators as observable characteristics 
of excellence established through consensus of professional judgement 
among practising teachers. The EQI-Art project provides documentation 
of those observable characteristics in i^tudent art which should be 
recognized and encouraged and which assist in the development of 
teacher and student expertise in art. It was designed as collaborative 
research involving eight expert art teachers (EQI-Art teacher researchers), 
four art specialists (EQI-Art Advisory Committee), and sixteen practising 
art teachers (field test and pilot teachers). The list of the participants in the 
EQI-Art project appears in Appendix A. 

Quality indicators of student art revealed through collegial exchanges of 
expert professional judgement are the focus of the project. Two primary 
considerations underlie the EQI-Art project research and development: the 
description of student artwork that demonstrates quality, and the 
development of teacher expertise in identifying quality work in the visual 
arts. 

Student Art Work 

Through the examination of student work we expect to document 
observable characteristics on which student performance should be 
properly judged. Curriculum guides and support materials typically 
contain significant detail regarding content to be mastered, skills to be 
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practised, activities to be engaged in, and resources to be made available. 
Expectations regarding quality outcomes, however, are practically non- 
existent, leaving one v >h the possible interpretation that quality outcomes 
do not really matter mucu, that the process of engagement alone is most 
important. Or, more positively, the impression is given that teachers have 
the ability to assess quality performance in art and do not need assistance 
in curriculum documentation. 

Teacher Expertise 

This second consideration is the core of student assessment in art, although 
the issues are somewhat different in elementary and secondary programs. 
In elementary schools there are few art specialists, and the majority of 
elementary teachers are teaching and assessing student performance in art 
courses with little expertise or training in art. In the secondary schools, 
there are proportionately more art teachers with expertise and training in 
the subject, and the elective nature of the school programs may encourage 
better performance simply due to the element of choice. However, the 
consistency of performance expectations, particularly through the 
collaboration of professional thinking, is not systematically provided for in 
support materials available to these teachers. 

The purpose of the EQI-Art project is to provide assessment devices that 
describe quality student performance in art, and do so in a manner that 
uses and develops teacher expertise. The project is founded on tlie 
principle that the articulation and communication of quality indicators for 
art production will lead to improved teaching behaviours and student 
learning. The following model depicts these ideas. 
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Tde model presents the art work as the central restilt of the practice A 
student sldU and knowledge, the application of student judgement, and the 
influence of teacher assessment skiUs and EQI models. The outer ring 
suggests the shaping interactions of teacher practice, reflection, curriculum 
content, and the" students' desires to create significant images. 
This model presents assessment as a rich and complex challenge that 
involves the teacher, the student, and the work. 

Table 1 on the following page outlines the project personnel, tasks and 
results from 1989 to 1992. A sample of EQI-Art project materials is 
included in Appendix B. 
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Year 1: Development 



In the first year of EQI-Art, teachers and advisors examined student art and 
identified characteristics of various levels of quality. Subsequently, several 
formats were developed for using this information in assessing an. There 
were four stages to the year's work: 1) identifying characteristics of 
quality, 2) raising concerns about predetermined characteristics, 
3) preparing a framework for art assessment, and 4) developing a 
prototype handbook. 

Identifying Characteristics of Quality 

During the first project meetings in 1989, EQI-Art teacher researchers and 
specialists reviewed class sets of student art and clustered the works into 
"below average," "average," or "above average" groupings. In each cluster 
of student art, the teacher researchers and specialists then attempted to 
identify quality characteristics. Although there was a lot of agreement and 
consistency in the clustering, it was far more difficult to identify 
"observable characteristics" that indicated the quality of individual student 
work. 

After reviewing many sets of student artwork, and categorizing each piece 
by its apparent quality, EQI-Art teacher researchers and specialists outlined 
some initial characteristics of quality that could be seen in student art 
works. The characteristics were: 

• significant use of elements of design - colour, line, shape, space, value; 

• application of the principles of design - "presentation," balance, 
organization, emphasis, repetition, variation, etc.; 

• demonstration of technical control - appropriateness, competence, 
creativity; and 

• evidence of thought - communication of a thought, emotion; 
complexity of work, novel use of symbol or medium. 

For each of the characteristics, the teacher researchers and specialists 
developed comprehensive lists of observations that could be made about 
student art. Teachers tried out these lists in tiieir classrooms and in project 
meetings. Teachers found the lists helpful in only a few instances. For 
example, tiie lists were helpful where the assigned art problem was very 
structured and limitations were imposed on media, size and technique. In 
some cases, however, the identified characteristics listed above were not 
seen to have any necessary connection with "quality" in the particular 
works. 
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It also seemed that a single set of characteristics was not sufficient for the 
variety of works students present. Therefore, a set of categories of 
artmaking — abstraction, working from observation or from memory, and 
personal expression or interpretation — were developed, within which 
more specific characteristics could be observed and outlined. These new 
lists were tried in classrooms and again, they proved to be of limited value. 

The lists were then restructured into the form of questions that required the 
user to respond regarding each of the characteristics that could be seen in 
student art work. For example, questions included: "What are the 
methods used to illustrate depth?" "How is your attention drawn to the 
centre of interest?" and "How is unity achieved?" When reviewed by the 
teacher researchers, it was decided that this structure would be useful only 
for those teachers who have considerable background in art, and many 
teachers do not. 

In attempting to provide sets of observable characteristics for several 
different types of artmaking, the group became frustrated with the 
constraints that were being established, and the unwieldy nature of the 
instruments. In addition, student art often did not strictly "fit" the four 
categories that had been defined. 

Raising Concerns about Predetermined Characteristics 

As a result of the concerns and frustrations with lists and questions, 
EQI-Art project committees felt that the checklists and sets of questions 
were not adequate to the task of assessing student artwork and the 
following concerns were raised: 

• a comprehensive list of everything one could see in art works was 
virtually impossible to create because of the variety in art; 

• if there were such a list, no teacher would have time to use it; 

• a checklist style might cause teachers to concentrate on elements in 
the art that, while easy to observe, contribute little to the overall 
quality; 

• a checklist approach might suggest that standardization is desirable in 
student artwork, and we wished to celebrate diversity; 

• there is great variety among artworks that are all considered "superior," 
or "below average"; 

• teachers need assistance in learning to observe art, rather than a 
technical tool that will have limited applicability; 
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• it is important to include the intended meaning of the student artist in 
the practice of art assessment; 

• to assess student artwork we must include the teacher intentions (or 
lesson presentation), and the classroom constraints; and 

• categories of art that would better capture the diversity of student 
artmaking were suggested: observation, interpretation, response, 
special communication of the self, and design problems. 

In response to these concerns, three basic criteria were established to 
provide a framework for observing quality indicators of student art: 

• RELATIONSHIPS of the elements and principles of art 

• HANDLING of the media used 

• MEANING or artist's intention in creating the work 
Five categories of art making were defined: 

• DIRECTED OBSERVATIONS: such works include contour drawings, 
still-life, drawings from observed objects, from nature, fiom art, and 
from designed objects such as toys, machines, or tools, etc. 

• NARRATIVE WORKS: these works show the student's response to 
ideas from language arts, social studies, sciences, etc. Any work that 
tells a story may be included in this category. 

• RESPONSES TO WORKS OF ART: such images come from 
experiences of art works. Teachers will sometimes have students 
study the work of a particular artist like Van Gogh, or a special art 
movement like Impressionism, and then have students use such styles 
to make new images. Included are art works that result from the study 
of the art of other cultures, 

• SPEQAL COMMUNICATIONS OF THE SELF: these are art works 
that reveal students dreams, fantasies, deep feelings about experiences 
and/or social problems, and memories. 

• DESIGN PROBLEMS: works in this category usually result from the 
challenge set by the teacher to create pattern, or to analyze balance, or 
symmetry, or to express ideas through special colour selections, i.e. 
warm and cool. 
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Preparing a Framework for Art Assessment 

The EQI-Art personnel rejected the use of checklists for assessing student 
art work because this structure was too limiting for the variety of artwork 
presented, and because the checklist "style" did not recognize the 
effectiveness of individual professional judgement. A narrative style of 
assessment was selected — one which modelled expert professional 
judgement based on connoisseurship and educational criticism (Eisner, 
1985). The expert judgement of practising art teachers was presented as a 
written critique individually prepared for each of the art works chosen by 
the project. The critique focused on the three basic criteria: relationships, 
handling, and meaning. 

The quality of a work of art is found in the completed work itself, but the 
work comes out of a set of conditions that give it additional meaning for 
the viewer. In the educational setting, an understanding of these 
conditions provides teachers with increased understanding about the 
process of making and assessing art. 

The identification of quality indicators in student art is practised within a 
school setting that consists of conditions — some of which can be largely 
controlled by the teacher and some of which are controlled by the student. 
These conditions are addressed in the EQI-Art project. 

In order to assess student works of art, teacher researchers wanted to 
know more about the conditions under which the art was produced: 

• what was the assignment? 

• how much background do students have/receive to do the 
assignment? 

• how much time was allowed? 

• what materials did the students have available? 

At the beginning of the project, EQI-Art teacher researchers brought in the 
work of their own classes and explained the conditions of the classroom 
assignment. Later, the teacher researchers at each grade level developed a 
joint lesson plan which was implemented and the results brought back to 
the committee. This latter plan was abandoned because the teachers found 
that lessons often had to be modified significantly because of resources 
available and student interests at the time. Quality artwork seemed to 
arise from lessons that engaged students in the shaping of the 
background information and in the interpretation of the art problem 
itself. The possible use of prescriptive lesson plans as a factor in the 
production of quality artwork was rejected as it did not fit the 
constructivist view of learning nor the belief in teacher professional 
judgement as the basis for educational decisions. 
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It was important, however, to know about the lesson that was presented in 
the classroom. With all samples of student artwork, we decided to provide 
a descriptive account from the teacher that included an explanation of the 
lesson, tiie teacher expectations, and the resources available (time, 
materials). A sample descriptive account from a grade 2 lesson follows: 



Lesson: 

The teacher challenged students to reflect that environments are altered 
by natural forces and to observe that shapes can suggest movement. 
Students were to observe examples of the changing patterns created by 
various objects moving through water and to note how light is reflected 
when water is moving. 

Using oil pastels, the students were asked to depict movement 
through water. Students observed many examples from art and 
from the teachers picture file. 

Teacher Expectations: 

The teacher expected that the students would: 

• imagine an animal swimming through water 

• create a two-colour oil pastel drawing showing how water 
changes when an animal swims through it 

• make patterns that show light and shadow on the surface of 
the water 

• indicate the movement of the waters wake 
Resources: 

Materials: Coloured construction paper, 12" x 24" 
Oil pastels 

Time: Two 45-minute class periods in the fall term 

Another condition for apprehending the quality of student artwork is the 
student intention in producing the art. 

Teachers who know their students well can speculate and make quite 
accurate inferences regarding student intentions much of tlie time. The 
EQI-Art teacher researchers felt that asking students to provide intentions 
directly would provide valuable information for the teachers in assessing 
the artworks and might also be valuable for students in forming and 
clarifying their ideas. Some students were asked to write about their 
intentions and some were audiotaped and videotaped. The written form 
seemed to be most useful. Figure 3 presents an example from the grade 2 
lesson described above. 
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Student Intention: 

/ drew a beaver swimming in a pond. Because he is moving his 
feet the water is going away in a wake. I drew his leg and had to 
draw these waves because he was moving. I drew some big 
circles far away. They are things in the water that isn't moving. 

Teacher Observation: 

The beaver shape is almost centred while the page is divided into 
three horizontal divisions giving the work a feeling of order and 
balance. The colour reversals in the large shapes — dark red 
centres, white centres, some with I :>rders, some without — are 
unusual and give variety and rhythm to these patterns. 

The series of small white shapes arranged in orderly, double- 
diagonal rows on either side of the beaver sharpen the contrast 
and create a sense of movement by leading the eye to the central 
beaver shape. 

The blue outlines and direction lines and the single-colour 
background make unity in the work 

Original is 12" x 18" 

Figure 4 
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Developing a Prototype Handbook 

Toward the end of the first year, the components of "quality indicators in 
student work" and "conditions of quality art production" were put 
together in the form of a prototype "Quality Indicators Handbook." The 
handbook reflects Eisner's model of connoisseurship and educational 
criticism (Eisner, 1985). The intention of the handbook is to illuminate 
the process of assessment through modelling expert art assessment 
practice. In order to provide this modelling, the handbook prototype 
contained teacher expectations, classroom conditions, and included colour 
reproductions of selected student work that expert art teachers would 
consider "superior," "above average," "average" and "below average." 
Each artwork was accompanied by student intentions and expert analysis 
focusing on relationships, handling and meaning. 

The EQI-Art teacher researchers used and critiqued the prototype model, 
making a number of suggestions regarding improvement. Three very 
important suggestions were implemented. 

First, it was recommended that we remove the qualifications "superior ... to 
below average," and present only those works that were of sufficient 
quality to allow extended critique regarding the quality present. Several 
factors were behind this recommendation. There was a concern that the 
presentation in published form of materials that would be held up as "poor 
examples" would detract from the emphasis on quality. A difficulty in 
copyright clearance in presenting student work in a negative light was 
also anticipated. Furthermore, the experience of the researchers suggested 
that by contemplating the organization of successful works, teachers 
would gradually build up a positive repertoire of quality indicators they 
could readily notice. 

Second, the teacher researchers recommended that some kind of summary 
list appear in the document to provide teachers with a vocabulary for 
addressing the characteristics of "Relationships," "Handling," and 
"Meaning." 

Third, there was considerable discussion regarding the value of providing 
handbooks for the specifically identified grades 8 and 11. At the junior 
and senior high school levels, there are often several grade levels in the 
same classroom, and in most schools only one teacher offers all of the art 
courses. In addition, the curriculum objectives within the junior high and 
senior high aie quite similar. It was recommended, then, that there be 
"junior high" and "senior high" handbooks, which could more widely used 
by art teachers. 
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These recommendations were implemented in the development of Quality 
Indicators Handbooks for grades 2 and 5, junior high, and senior high 
school grades. Each handbook contained several teacher explanations of 
art lessons related to the curriculum at that grade level. With each lesson 
explanation is a set of colour reproductions of quality student art. Each 
reproduction is accompanied by "student intentions" and "expert 
observations" of the quality indicators that can be seen in the artwork. 
The format of the handbooks invites the users to become involved in the 
process of "seeing with expert eyes'' and to model the same process in their 
own assessment practice. 

Year 2: Field Testing 

In the second year of EQI-Art, teachers and advisors developed and 
carried out a plan for the field testing of the EQI- Art handbooks in local 
classrooms. There were four stages to this year's work: 1) field testing the 
handbooks, 2) responses from the field-test teachers, and 3) revising 
the handbooks prior to p'iOtuig. 

Field Test of the Quality Indicators Handbooks 

Eight successful, professionally active teachers from the two Calgary 
boards were selected to field-test the Quality Indicators Handbooks in 
their classrooms during February and March 199L These teachers were 
given a brief inservice and invited to use the handbooks in many different 
ways in their art teaching and assessment. 

The EQI- Art field test was designed to encourage practitioners to use and 
examine the Quality Indicator Handbooks within the practice of their 
teaching situations in order to validate the research of the committees. The 
handbooks are intended to be used as a resource for the guidance of the 
practice of assessment in the learning environment. 

The stated objectives of the field test were to: 

• identify the strengths and weaknesses of the EQI-Art Quality 
Indicators Handbook (QIH) as a teacher resource; 

• collect information from teachers who would be defined as "specialist" 
and "generalist" with regard to response to and use of QIH materials; 
and 

• collect data about classroom conditions that contribute to the 
development of quality student art. 
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Field test teachers were involved in the following tasks: 

• Field test teachers prepared a written assessment on a work of student 
art prior to the use of the QIH, then prepared a second assessment on 
another work at the end of field testing. These two assessments were 
analyzed by the EQI-Art advisors. 

• The selected teachers field tested the initial version of a survey, 
"Teacher Survey of Assessment Attitudes and Skills/' designed to 
answer questions about relationships between the charactenstics of 
experience, confidence, and expertise in art assessment, and the 
belief systems of art teachers. Results are found in Appendix B. 

• Field test teachers responded to the QIH materials by writing directly 
in the handbooks to suggest changes, additions, deletions to any parts, 
and by documenting classroom practice using the QIH. 

• If teachers used the QIH materials in evaluative procedures, they were 
asked to share the manner or structures developed for this purpose. 

• If teachers involved students in using the QIH materials — for example 
in self-assessment or peer assessment — they were asked to document 
the use in written accounts, audiotape or videotape. 

Responses to the Field Test 

At the end of the field test, information was collected on teacher attitudes 
and skills, uses made of the handbooks, and concerns teachers had in 
implementing the assessment practices. Responses from the field test 
teachers included perceived effects of the use of the QIH in the classes, 
recommended revisions to the handbooks, and suggestions for inservicing 
pilot teachers for next year. 

A few of the field test teacher comments related to tlie use of the 
handbooks follow: 

• I found that when they [students] were doing their artwork — they 
were standing back and looking at how they could make changes 
and thinking about what they could do to improve their wort 

• / referred to the pictures in the handbook and we discussed them in 
terms of quality indicators, and they [students] would do whatever it 
was that we were doing ...but they would really tune into the things 
that we had been discussing. 
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• / thought the intention [question] was excellent for the student ... / 
think it made the kids focus in, at least in junior high, on the 
objectives [of the lesson] afid *hey were conscious of this 
throughout, 

• I think probably technically they [students] would look for things 
more easily. They would really start to look at compositions and 
they would realize ... the influences they've had from other artists ... 
but maybe that's just because Im prompting them more too. But I 
would say ^yes* it's [EQI-Art] making a difference. 

The field test teachers were also involved in the process of looking at 
changes in their own assessment practice. Prior to beginning to work with 
the handbooks, each teacher was asked to write a critique of a student art 
work. At the end of the field test, teachers were again asked to write a 
critique, considering a different student piece. A comparison indicated an 
increase in specificity. In the earlier example, a teacher might write "... a 
good use of colour,* but in the second example, write "... blue adds 
definition to the figure and also gives the snowman a ^ cool feeling,'' 

Comments and recommendations provided by the field test teachers were 
used to clarify some of the instructions in the handbook, to add some 
insights to the art critiques included, and to reformat some of the materials 
for easier classroom use. The field test response of teachers and students 
to the EQI Quality Indicators Handbooks was enthusiastic. Improved 
practice was noted by the field test teachers themselves in a majority of 
cases. 

Revisions to the Quality Indicators Handbooks Prior to Piloting 

The initial handbooks used for field testing included a limited number of 
samples of student art, representing only two or three of the five art work 
categories identified by the EQI-Art project. Given the positive response 
to the style and contents of the handbooks, the project team decided to 
complete the presentation of all categories for the pilot year. 

To obtain the art works necessary for this expansion, an invitation was 
sent to all art teachers in the two school districts to submit sets of student 
work for consideration. A significant collection of work was submitted for 
consideration. The EQI- Art teacher researchers and specialists who 
prepared the first edition of the handbooks prepared die pilot editions 
during the month of June 1991. 

Initial handbooks contained only one grade level, or, as in junior and 
senior high, only one set of grades. Field test teachers requested that all 
grade levels be included in one handbook so that they could be used to 
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illustrate a development of art practice and performance throughout the 
grades. The field test teachers suggested that the opportunity to look at 
the collection of work from all grades would benefit students as well as 
classroom teachers. 

Consultants recommended that the project leadership reconsider die 
decision not to represent a variety of qualities, from below average to the 
truly outstanding. The consultants suggested that the handbooks would 
be more effective if they help generalist teachers to "recognize poorer 
performance along the various dimensions'' (Maguire & Rogers, 1991). The 
expert teachers' experience, however, indicated that a repertoire of good 
examples of student work encouraged improved performance. Teachers 
and students in the pilot group appeared to learn more from the 
consideration of successful works rather than from the analysis of inferior 
or incomplete works. In addition, copyright laws do not permit the 
presentation of individual student work in a negative light. Consequently, 
the project leadership decided to stay with the original decision, to present 
only works which were of good quality for each grade level. Negative 
exaniiples of student art work were not included. 

However, at the beginning of every handbook, there would be a listing of 
the most obvious quality indicators — relationships, handling and meaning 
— with characteristics that can be observed. These outlined 
characteristics should assist a non-expert art teacher in identifying the 
dimensions of various qualities as well as in developing a vocabulary to 
use in providing art criticism. 

Year 3: Pilot Testing 

At the end of the second year, the EQI-Axt project advisors revised the 
initial handbooks by increasing the number of art samples, and by 
including all grade levels in one volume. In order to increase the 
awareness of student thinking and to capture teacher reflections on art 
assessment, the art advisors incorporated reflective journals for both 
teachers and students into the pilot plans. 

In the third year of EQI- Art, teachers pilot tested the handbooks in local 
classrooms. A variety of activities helped gather information about the 
value and usefulness of the handbooks. These two sections will describe 
the activities: 1) pilot testing the handbooks, and 2) information 
gathering and analysis. 

Pilot Testing the Handbooks 

From September 1991 until the end of January 1992, eight practicing art 
teachers from CBE and the CCBE participated as pilot teachers. As in the 
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field test, the selected teachers were successful and professionally active, 
and participation was on a voluntary basis. 

The objectives of pilot testing were to: 

• identify changes in the assessment skills and attitudes of teachers 
using the EQI-Art Quality Indicators Handbook (QIH); 

• identify changes in the assessment skills and attitudes of students 
using the EQI-Art QIH; 

• collect information from teachers who would be described as 
"specialist" (teachers with a university degree in the field of art or art 
education) and "non-specialist" regarding their response to and use of 
QIH materials; 

• collect information about pilot classroom conditions that support or 
hinaer the development of quality student art; 

• provide an indication of the changes in the quality of student art work 
that may be expected as a result of teacher and/or student use of the 
QIH; and 

• identify the strengths and weaknesses of the EQI-Art QIH as a teacher 
resource. 

Pilot teachers v/ere familiarized with the background of the EQI-Art 
project and the development of the handbook during a day-long inservice 
session. Pilot teachers were asked to: 

• use the handbook as a model to practice one or more assessment 
activities in their classes; 

• have students use the handbook in classroom activities; 

• use the model of the handbook to make one or more collections of 
their own, based on a classroom activity completed during the pilot 
period; and 

• critique the handbook by writing in the book itself, by commenting in 
the reflective journals, and by reporting concerns to the project 
advisors. 

Pilot teachers were provided with instruction and guidance about writing 
journal entries. They and their students were asked to maintain a journal 
of their reflections regarding the use of the handbook. The organizing 
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questions for the journals were: "What did you learn from your use of the 
handbook today?" and "What questions occurred to you in your use of 
the EQI materials?" 

During the pilot period, the art advisors visited all pilot sites to see how the 
work was going and to review the teacher and student journals. 

Information Gathering and Analysis 

Information gathering occurred prior to, during, and following the piloting 
experience. The information and analysis is reported here in four sections: 

a. Identifying change in teacher attitudes and practice 

b. Examining the use of teacher and student joumals 

c. Parent responses to the EQI-Art pilot and handbook 

d. Pilot teacher "Conclusions" and "Recommendations" 

a. Identifying change in teacher attitudes and practice 

Pilot teachers completed a survey of attitudes and practice on two 
occasions — at the beginning of the pilot period, and at its conclusion. 
This survey was developed to identify changes in beliefs, attitudes, 
and practices relating to the use of QIH. The EQI- Art Pilot Teacher 
Survey is found in Appendix B. The survey provided demographic 
data about experience, background, and teaching assignment, as well 
as information regarding the beliefs and practices of teachers 
regarding art teaching, assessment, and student artmaking. The survey 
questions were also designed to collect information regarding Eisner's 
(1985) ideas of connoisseurship and criticism: "expertise," 
"confidence," and "use of subjective methodologies in art assessment." 

Analysis: Demographic and descriptive data from the survey are 
reported in Appendix B. Analysis by SPSSX yields only 
a few statistically significant changes in teacher attitudes 
and practice between the two survey administrations, 
and these changes are not consistent in the direction of 
more or less confidence or expertise. The findings are 
not reported here because we feel the results are of 
limited use, given tiie small number of participants (1 1), 
the limited time (4 months) and the lack of reliability 
testing of the survey. 

Pilot teachers also provided feedback to the art advisors both 
verbally and in written form at tiie final pilot meeting. Almost all pilot 
teachers commented that there were changes in their practice as a 
result of EQI. These changes included plans for using journal writing 
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in future classes, awareness of changed viewpoints, needing to listen 
more to student intentions, and learning new ways to talk to parents 
about student progress in art. Some of their written comments foil jw. 

• [I] discovered how much students enjoy viewing other students' 
work. In fact, that is when it is easiest to gain and maintain 
class attention. They loved hearing the artist intentions. 

• / credit art with being a higher level cognitive activity than 
before. As children wrote about their work they identified what 
must have been going through their minds subconsciously and 
brought it to a conscious level 

• 1 found that because of my involvement with EQI my way of 
teaching has changed — it has also carried over to other 
subject areas as J have discussed and worked with my class on 
ways to produce "quality" work. They and I both look at what 
they are doing differently — what makes this quality work? 
what are they happy with, and why? what might they like to 
work more on? and what was difficult for them? 

h. Examining the use of teacher and student reflective journals 

Pilot teachers were provided with journals for their own use and for 
the use of their students at the start of the pilot period. During the 
course of the pilot, art specialists from the two school systems visited 
all of the pilot classrooms, taking time to review teacher and student 
journals and to write in them when appropriate. At the completion of 
the pilot period, teachers brought these journals back to the project 
advisors for analysis. Most of the pilot teachers commented 
positively on the use of journals, both for themselves and for their 
students. Most teachers identified a connection between student 
writing and thought processes that enhanced the quaUty of student 
work. 

Analysis: Two experienced art teachers currently enrolled in 
graduate art education programs at the University of 
Calgary were engaged to carry out an analysis of the 
journals used in the EQI-Art pilot. These teachers had 
experience in journal analysis, and had some familiarity 
with the EQI project. Their conclusions follow. The 
complete report is found in Appendix B. 

In conclusion, we saw the journals as holding the possibility for 
being an excellent means to foster artistic learning. The opportunity 
for teachers to reflect on their own practices was described by many 
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as being invaluable," Their constant questioning and striving to 
understand the project was made clearer through their reflections, 
observations and feelings about their art classes and students. We 
saw the place of the teacher as pivotal in regards to facilitating the 
use of the journals in an approach that promoted a continuing and 
evolving discourse. The students journals fostered reflection on 
their art learning, and developed a heightened sensitivity to elements 
within a whole. It is hoped, therefor'^,, that this encouraged quality 
improvement in student art work. 

c. Parent responses to the EQI-Art pilot and handbook 

Parents with students in the pilot classes were contacted and invited 
to discuss the project experienced by their children. Meetings were 
held in the pilot schools with small parent groups. 

Analysis: Written and oral responses from parents were positive 
towards the project as they understood it, and they 
expressed appreciation for being invited to respond. 
Sample comments follow. 

• / think the art handbook is a great idea. It helps to give the 
children an idea of what is expected on certain projects. It 
also helps to set out guidelines so the teacher knows what to 
look for I think it will help the children to take pride in their 
art work and put more time and effort into their projects. 

• Good concept. Shows children what is expected. The written 
paragraph gives a child a chance to express their idea or 
intention. 

d. Pilot teacher conclusions and recommendations 

During the final meeting of the pilot teachers, we asked the teachers 
to respond in writing to the headings listed above. We collected 
these responses and recorded discussion regarding the pilot 
experience. 

The conclusions reached by the pilot teachers often refeired to the 
value of "talk," "language" and "vocabulary" as important aspects of 
focusing student attention on the art problem, getting students to 
articulate their knowledge about art, and sharing understandings of 
quality in art. Another main theme was the opportunity to see and 
hear what colleagues had to say about art assessment. Many 
teachers wanted an extension of the pilot meetings, and expanded 
versions of the EQI handbook. 
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• / think the most positive aspect of the project was having the 
students write down their thoughts and intentions about their 
work. It focused their ideas, concerns and observations about 
their work, 

• [The EQI-Art Handbook] provides concrete examples of how to 
talk about specific pieces of art. Teachers may be familiar with 
terms such as pattern, unity, etc., but not know how to use these 
in discussion. By simply reading what other teachers have 
written about children's art makes it easier to incorporate these 
ideas into classroom practice. 

The teachers had many recommendations for future developments, for 
inservice, for other ways to use the book in classes, and for using 
joumals as an integral part of assessment. Some comments follow. 

• Inservice of some form is essential to use the book as a resource. 

• It is really important that teachers and students have the time 
and opportunity to reflect on their work. The use of journals is 
an important aspect of this. It is just as important that teachers 
and students be given the opportunity to dialogue with each 
other and their peers — EQI certainly heightens one's 
awareness of the possibilities of art education and how to more 
accurately and fairly assess student work. 

Follow-up activities suggested for teachers, for schools, for school system 
art specialists, and for Alberta Education included: that handbooks be 
made readily available, that time and support be given to teacher meetings 
about assessment, and that the EQI project people have a big celebration 
to commemorate our work together. 

Pilot teachers were asked to critique the handbook and to bring editorial 
and structural suggestions to the final pilot meeting. A thorough review of 
the entire handbook was held during the meeting and all comments and 
suggestions were considered. The EQI-Art advisors revised and edited the 
entire handbook during February and March 1992. The desired changes 
included: an expanded introduction to the handbook; suggestions for the 
use of the handbook in classrooms; suggestions for provoking discussion 
regarding quality in art; and practices that stimulate journal use. Revised 
handbooks were provided to all teachers and advisors who worked on the 
project, to teacher resource departments in the two school boards, to 
university personnel for use in teacher preparation, and to Alberta 
Education. 
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Findings and Conclusions (Outcomes) 

The EQI-Art researchers, both teachers and specialists, understood from the 
outset that the project required a lengthy period of development and trial 
and error. This indeed proved to be true, as we tried and discarded many 
different approaches during the initial year and a half. Our committee 
process was to question, to research, to intuit, to test, and to revise our 
materials and our ideas about assessing quality art. The final handbook 
represents the wisdom we collectively shared regarding the assessment of 
quality in student art. 

The art advisors considered information collected from all sources and in 
many forms in preparing the following conclusions about the EQI-Art 
project: 

• Quality assessment practice supports improved art making. 

• Written and/or spoken student responses, reflections and thoughts 
illuminate the art work. 

• Quality an assessment is a shared experience between teachers and 
students. 

• Common vocabulary shared by teachers and students affects 
assessment. 

• Collection of quality art exemplars emphasizes diversity as an attribute 
of a quality art program. 

• Informed professional judgement benefits from collegial work. 

• Assessment takes time: time for students to prepare a quality outcome 
and for teachers to thoughtfully reflect and respond. 

• EQI handbook exemplars provide a way to share expertise and 
vocabulary, and to develop connoisseurship. 
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CHAPTER 4: EQI-MATH 



Introduction to EQI-Math 

The most important goal of mathematics instrucaon in Albena is the 
development of students' abilities to solve problems. The General Learner 
Expectation for mathematics is that: students are confident and 
increasingly competent in solving problems as they develop and 
demonstrate understanding of mathematical concepts, relationships and 
procedures (Alberta Education, 1992, p. 3). 

Problem- solving abilities and attitudes are also at the core of descriptions 
of numeracy, or mathematical literacy. The lack of mathematical literacy 
has been identified as an economic liability for a nation, a personal 
limitation, and a source of systematic discrimination for women and 
minorities (National Research Council, 1989; National Council of Teachers 
of Mathematics, 1989). 

One of the prime documents recommending change. Curriculum and 
Evaluation Standards for School Mathematics (National Council of 
Teachers of Mathematics, 1989), describes standards for mathematics 
literacy in the 1990s and provides guidelines for the revision of curricula 
and assessment toward this literacy. There are five goals for students: 

1) learning to value mathematics 

2) becoming confident in one's own ability to do mathematics 

3) becoming a mathematical problem solver 

4) learning to communicate mathematically 

5) leaming to reason mathematically 

These considerations clearly extend the expectation of mathematics 
instruction from the acquisition of specific skills and processes through to 
positive attitudes toward and the use of mathematical reasoning in many 
different life situations. This expansion of expectations regarding problem- 
solving skills and dispositions to problem solving has been accompanied 
by some change in teaching and testing methods in schools, and intense 
interest in developing new methods in response to the need for change. 

The EQI-Math Project involved practicing mathematics teachers and 
mathematics subject specialists in a search for new methods to assess 
student achievements in problem solving. Table 2 on the following page 
oudines the activities of the EQI-Math project from 1989 to 1992. 
Appendix C contains a sample of the project materials. 
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Year 1: Development 



Quality indicators are observable characteristics of excellence established 
through consensus of professional judgement among practising teachers. 
Problem solving is the application of known mathematical knowledge to 
an unfamiliar situation so as to arrive at a solution (EQI Project definition). 

The expectations and objectives of mathematics programs direct attention 
to students applying math skills and processes and approaching the use of 
mathematics with self-confidence, a positive attitude about the efficacy of 
math in understanding real life situations, and enthusiasm. The indicators 
of problem solving therefore apply to these aspects of student outcomes as 
teachers observe them in classrooms. 

During the first year of the EQMMath project, teachers and advisors 
examined many aspects of mathematical problem solving, developed sample 
problems and prepared materials for use in recording observations about 
student problem solving. The four main areas of activity for Year 1 are 
presented in the following sections: 1) identifying student dispositions; 
2) specifying problems for use with students; 3) developing holistic 
marking; and 4) documenting student iderstanding. 

Identifying Student Dispositions 

We began our quest for quality indicators of mathematical problem solving 
with a team of 14 successful, professionally involved mathematics teachers 
(EQI-Math teacher researchers) and 5 mathematics subject area specialists 
(EQI-Math Advisory Committee) from the Calgary Board of Education 
and the Calgary Catholic Board of Education. Personnel involved are 
listed in Appendix A. 

The selected teachers and advisors began their work by exploring the 
concept of quality in matiiemaucal problem solving. It was apparent that 
experienced mathematics teachers identify successful problem solvers by 
their dispositions or personal characteristics, in addition to their possession 
of particular math skills and success in solving math problems. Initial 
meetings of the EQI-Math committees centred around discussions of what 
defined good problem solving, and how to identify good student problem 
solvers. The identification of good problem solvers led to the development 
of a list of characteristics or traits that described student "dispoj=itions" 
(The National Council for Teachers of Mathematics): 

Disposition ref(.'.rs not simply to attitudes but to a tendency to 
think and act in positive ways. Students' mathematical 
dispositions are manifested in the ways they approach tasks — 



EQI - Art & Mathematics 4 0 Calgary Board of Education 

Calgary Catholic Board of Education 



whether with confidence, willingness to explore alternatives, 
perseverance, and interest — and in their tendency to reflect on 
their own thinking (NCTM, 1989, p. 233). 

Initially, teacher researchers recorded observations of strong and weak 
problem solvers in their classes to obtain descriptive comments about 
problem-solving dispositions. These records were in the form of informal 
"anecdotal records." Teachers found this task to be difficult because there 
was so much that could be observed, and decisions about what to include 
or not include in the anecdotes were complex. The teacher researchers 
shared their experiences with one another and discussed anecdotal records 
for the students in their classrooms. As a result of these activities, ten 
dispositions were identified, and descriptions of each disposition were 
developed from teacher observations. 

The initial lists of dispositions prepared by teacher researchers at both 
elementary and secondary grades were very similar, so it was decided to 
proceed with one list of dispositions for students at all grade levels. The 
teacher researchers agreed that although not all of the dispositions would 
be displayed by any one individual student, a student who was a good 
problem solver would display at least several of them. Appendix C 
contains the original ten problem-solving dispositions and descriptions. 
Seven of the ten dispositions described are similar to those outlined in 
Curriculum and Evaluation Standards for School Mathematics (NCTM, 
1989, p. 233). 

Specifying Problems for Use with Students 

The dispositions towards problem solving are developed over time as 
students experience classroom mathematics activities. An iixiportant part of 
activities to identify quality indicators is the context for students to display 
their dispositions and their problem-solving skills. Student problem- 
solving dispositions were observed in the classroom context where 
students were engaged individually and in small groups with mathematics 
problems. 

Teacher researchers began by selecting student mathematics problems to 
conform to the curricular demands of each grade studied (grades 2, 5, 8, 
and 10) at the time of the school year the problems were to be 
administered. Problems were identified for each strand of the curriculum 
and for non-routine, challenging situations. Teachers used personal 
knowledge, resources available in the two school systems, and pubUshed 
sources for problem ideas. 

Attention was paid to presenting a variety of problems from the 
mathematics curriculum strands, and to encouraging substantial student 
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responses. No attempt was made to control the nature of the problems at 
each grade level, and teachers prepared a large "bank" of problems from 
which to choose. 

The teacher researchers considered the format of the problem statement 
and the student response sheet to be important elements. A format was 
selected for each grade level, including a prompt for students to show all 
their work, a response structure (especially for grades 5 and 8) and some 
statement of affect from the student (How did you feel ... ?). Samples of 
these problems are found in Appendix C. 

Developing Holistic Marking Criteria 

The EQI teacher researchers began the exploration of scoring problems by 
reviewing problem solutions provided anonymously by other teachers. 
ITiese solutions were used in several ways: to identify quality in student 
responses; to develop some common expectations for student problem 
responses; and to gain an appreciation of the variety of problem-solving 
responses and interpretations there could be. 

The teacher researchers used marking practices that varied from holistic or 
impressionistic, related to the overall nature of student performance, to 
analytic or rubric-based, including specific predefined requirements that 
reflect the expert opinion regarding the important elements of a problem. 
The teachers reviewed a wide variety of systems used in holistic marking 
prior to developing some formats for scoring the problems used in the 
EQI-Math project. An holistic structure, representing each problem 
solution as either "A" inadequate, "B" adequate, or "C" competent were the 
categories used during the first year. 

Documenting Student Understanding 

Teacher researchers continually considered the problem of understanding 
the thoughts of students that were not expressed in their written solutions. 
They attempted to improve the fullness of student responses in various 
ways: by preparing a structured response form for students in grades 5 
and 8; by using the encouragement "Please Show and Explain All Your 
Work" on every response sheet; and by providing lots of paper for the 
student response. Teachers also considered the use of journal entries or 
learning log entries to clarify student thinking. 

The project activities in the first year considered: the presence in students 
of positive dispositions towards problem solving; the quality of math 
problems; marking approaches for the written responses; and discovering 
more about the thinking of students. The activities of the second year 
built on this work. 
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Year 2: Field Testing 



The project activities in the second year addressed five factors that 
appeared to contribute to quality problem solving in mathematics: 
1) student problem-solving dispositions; 2) selection and presentation 
of problems; 3) holistic scoring scales; 4) student attitudes; and 5) 
conditions of mathematics learning in the classroom. EQI-Math 
teachers and advisors prepared and field-tested materials and practices as 
they worked together to extend their knowledge about each of these five 
factors. 

Student Problem-solving Dispositions 

Teacher researchers documented classroom observations of the 10 student 
dispositions on several occasions. Students were observed in mathematics 
classes (not only during the completion of the EQI assigned problems) and 
a record was to be made of whether the disposition was "present'* (+), "not 
present" (-), or "not observed" (0). A sample Disposition Checklist is found 
in Appendix C. 

The teacher researchers found that using the disposition checklists was a 
difficult activity, First, many teachers found the number of observations to 
be made cumbersome. 

• / really have found the inventory sheets and disposition sheets for 
the w^hole class time consuming and difficult to do so early in the 
school year. One problem may show only one or two of the 
dispositions given. So it seemed to take a lot of problems in order to 
assess all dispositions. 

• [There are] too many dispositions to track for each student. 

Second, many teachers were personally uncomfortable with the process of 
using the checklist and the inferences that were made from the observed 
behaviours, 

/ was overwhelmed and frustrated trying to complete the dispositions 
— 1 like to be accurate and somehow see this [impression] as being 
written in stone. 

Some of the dispositions — I felt I was guessing on and would have 
difficulty giving valid reasons why I had given the child a or a 

• I feel the disposition sheets are not valid at this point. Vmjusi 
beginning to know my students and it is only now that I feel 
comfortable making comments on the students. 



EQI - Art & Mathematics 



43 Calgary Board of Education 

Calgary Catholic Board of Education 

5.:J 



However, teachers did find that the exercise of completing the disposition 
sheets allowed them to discover interesting and valuable information 
about individual students. 

• rm finding it dijficult to be objective. However ^ I do find that 

knowledge about the children, how they approach tasks, what kind 
of a communicator they are, is important. 

During the second year, the teachers were also provided v *th "Anecdotal 
Record" sheets to record observed behaviours for each of Jie 10 
dispositions for two students. Each teacher was asked to record 
observations of one "good problem-solving" student, and one "weak 
problem-solving" student. An example of one of these records. Inventory 
of Mathematical Disposition Experiences for .... appears in Appendix C. 
This was a challenging task, but all teacher researchers completed at least 
two of these records for students in their classes. These records were 
collected into grade group descriptions, and summarized for students at 
each of the grades. 

The anecdotal record activities proved to be very illuminating for the 
teacher researchers and also for the project advisors. There was a variety 
of interpretations of the dispositions, and variation in the clarity of 
description provided by the teachers. Some teachers prepared lengthy 
descriptions of student behaviours — "Her communication about doing 
math problems is at afunctional level For example, first you do this 
and then this. She doesnt seem to look for concepts,'' and other teachers 
recorded very brief comments, such as "Student communicates well" 

Because identifying student dispositions and recording descriptions 
presented a number of difficulties for the teacher researchers, some 
revisions to the materials and procedures were made. Three approaches to 
reformulating the identification of dispositions were pursued: 

a. Consolidating the dispositions to be identified. Factor analysis 
using a principal components method with iterations was carried out 
on the checklist data prepared by the teacher researchers during the 
second year of the project. One significant factor was obtained 
which was highly related to other factors. For the pilot year, in order 
to conceptualize this factor, dispositions were reduced to five and 
descriptions altered and refined. The five dispositions and their 
descriptions, "Mathematical Dispositions," are in Appendix C. 

b. Reducing the number of observations required of teachers. Pilot 
teachers were asked to record observations of the five problem-solving 
dispositions twice during the pilot period — early in September and at 
the beginning of December, toward the end of the pilot, 
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c. Restructuring anecdotal record forms and procedures* Pilot 

teachers were given inservice with a highly structured format prior to 
the pilot. Three records were made during each problem assignment. 

Selection and Presentation of Problems 

The teacher researchers selected eight problems to field test with their math 
classes. The student responses were rated by the teacher, then by one 
other teacher at the same grade level. Teachers at each grade level met 
several times during the field test to discuss concerns and new learnings. 

Field testing the problems provided information to teacher researchers on 
the following issues: 

a. Wording of problems. The teacher researchers discovered that 
several problems were misunderstood by students partially as a result 
of the words used. Problems were reworded by the advisors. 

b. Answer formats. Advance organizers, in the form of "boxes" for 
parts of the problem-solving process for students at grades 5 and 8 
received mixed reviews from EQI-Math teacher researchers and 
advisors. Several teachers found the boxes very helpful in organizing 
and encouraging the work of their students, while others observed 
that the boxes inhibited student responses. The advisors decided to 
remove the boxes from the problem solution sheets used in the pilot. 
Problems for piloting are printed on 1 1" by 17" sheets, folded to a 8.5" 
by 1 1" format. The problem sheets provide about three regular sized 
sheets of space for the student solution. 

c. Time allowed for completion of problems. During the field testing 
of problems, teachers decided on the appropriate time limits in their 
own classes. Although the time allowed varied among classes, this 
practice did not seem to pose any problems for the project and was 
continued in piloting. 

d. Individual or group administration. Teacher researchers selected a 
method of problem presentation that suited their classes and teaching 
styles. Some encouraged students to confer with partners or in small 
groups and some requited the students to work strictly by 
themselves. All students prepared a response sheet for each problem. 
Although there was some discussion regarding the "fairness" of using 
collaborative processes, the advisors decided not to restrict how a 
teacher presented the problem. The pilot teachers were asked to 
record if collaboration was allowed with each problem. Teachers 
were reminded that the EQI-Math project materials were not designed 
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to compare individual achievement for evaluative purposes, and 
therefore did not require the strict control of conditions. 

Consultants working on the provincial EQI project encouraged the 
EQI-Math advisors to review and revise problems as necessary, with a view 
to producing a set of guidelines for the construction of "good" problems. 
Rather than keeping EQI-Math problems confidential, consultants 
encouraged the development of many problems and the preparation of 
problem constmction guideUnes. It was also determined that six problems 
would be sufficient to provide qualitative samples during piloting. 

Holistic Scoring Scales 

The teacher researchers field-tested eight problems at each grade level. 
They were asked to write out their holistic scoring criteria, then score each 
problem solution either "A" inadequate, "B" adequate, or "C" competent. 
This letter rating was marked on the back of the student response. A 
second teacher re-scored the paper, using the same three letter markings. In 
some cases, a third marker was used to provide an additional rating of the 
student response. Teachers did not discuss their interpretations of the 
holistic criteria at length prior to the scoring exercises. 

After the scoring process was completed, teachers were asked to explicitly 
define their interpretations of the holistic scoring criteria. These were 
collected and then summarized. 

Consultants with the provincial EQI project reviewed the holistic scoring 
processes and criteria, and recommended that the project consider 
structuring categories that reflected levels of elaborated thought — from 
very minimal or simple, to highly elaborated, abstract conceptions. 
EQI-Math advisors were encouraged to provide more than three 
categories of holistic ratings — perhaps by adding a rating of "elegant," or 
by adopting a five point scale. 

Subsequently, the EQI-Math advisors prepared ^ new holistic scale. The 
characteristic that discriminates between the first three levels of scoring is 
the degree of progress made toward a solution, from partial to complete. 
The fourth level, "elegant,'' is self-explanatory. Advisors also changed the 
depiction of the holistic categories from scoring criteria to response 
criteria, A copy of the Holistic Response Criteria is found in Appendix C. 

Student Attitudes 

In order to understand better how students respond to problem solving, 
the teacher researchers decided to include a measure of student attitudes 
with each of the EQI problems. The attitude question was placed at the 
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bottom of each answer sheet. During the field testing, each grade level 
used a different attitude response format. 

The grade 2 format included three faces: "happy," "neutral," and 
"unhappy." Data analysis indicated that student attitude related to problem 
ratings when the problem was difficult. However, it was interesting to 
note that a student who received a rating of "A" (inadequate) on a problem 
that is easy or of mid-range difficulty was just as likely to be "happy" as a 
student who received a rating of "C" (competent). Put in anotiier way, a 
student who received a rating of competent is just as likely to be 
"unhappy" as a student who received a rating of inadequate. 

Grade 5 students were piovided with the prompt: "How I feel about this 
problem? Explain." with lines provided for the response. Grade 8 students 
were asked: "Circle words tiiat tell how you feel about this problem or write 

your own words. Easy Challenging Confusing 

Fun or ." At tiie Grade 10 level, students were asked to: 

"Circle words that tell how you feel about this problem or write your own 
words. Boring Challenging Confusing Fun 



Asking students about their feelings towards a problem was thought to 
help teachers prepare more interesting or appealing problems. Another 
dimension of student attitude is the feeling a student has after completing a 
problem. To obtain this information, advisors decided to ask students to 
respond to a statement that refers to how tiiey feel after attempting to 
solve the problem. 

The EQI-Math pilot problems included two statements: 
The word I would use to describe this problem is: 

Boring Challenging Confusing Fun 

After trying to solve this problem I feel: 

Discouraged Satisfied Frustrated Happy 



Conditions of Mathematics Learning in the Classroom 

Teacher researchers were asked to assist in describing their classroom 
mathematics environment by responding to the statement: "Describe what 
is happening in your math classroom." This description was to be made at 
die time of presentation of each EQI problem. Each teacher researcher 
provided eight descriptions of the classroom environment No specific 
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format was prescribed for the responses, although teachers were 
encouraged to talk about such elements as the curriculum emphasis, style 
of teaching, and unique classroom features they thought might influence 
the problem-solving results. 

Some of the teachers used the response to describe lesson topics: 

• Geometry has been the main focus - symmetry, constructing 3-0 
shapes, I-D shapes. 

• Free explorations of manipulatives. Group problem solving every 
day. Group graphing (i.e. birthdays). Pattern problems (Guess my 
rule: shoes with laces or without laces). 

Other teachers described student organization: 

• We have formal group instruction in math in the morning (with or 
without manipulatives, depending on the developmental level of the 
children). In the afternoon we have learning centres in which the 
children practice and extend their skill development. The children 
sit in groups of five. We have some group solving of problems as 
well as individual. We will also be working in pairs, but have done 
little of this yet. 

Still other teachers focused on student responses to the mathematics 
instruction: 

• Students are beginning to enjoy problem solving. They begin 
problems in groups and understand never to leave a piece of paper 
blank. Some are becoming real risk-takers, while a few want to 
always play it safe and find the ''correct" answer. 

• Students are becoming better able to cooperate and work in small 
groups as the school term continues. 

Teacher researcher responses varied in comprehensiveness, style, and 
detail. There was little in common among the responses to characterize 
classroom styles. 

A very interesting finding for the EQI-Math advisors was the indication 
that for some of the teacher researchers, problem solving is still a "Friday 
event" rather than an integral part of the daily mathematics instruction. 

Teacher researchers were asked to respond to the statement: "How did 
you present this problem to your students?" Again, no specific framework 
was given for the responses and again, both the quality and the nature of 
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the records varied considerably. This information identified whether or not 
students collaborated with one or more classmates while working on the 
EQI problems and provided additional background about the classroom 
environment. 

After working with the EQI-Math project for two years, the teacher 
researchers were asked to respond to a set of questions regarding their 
experiences in working with the project and how the project affected their 
teaching practice in the area of mathematical problem solving. 

The teacher researchers were unanimously positive about the experience 
in working with the project, most often citing the opportunity to work 
with colleagues, the learning about students, and the exposure to new 
ideas. Their concerns with the disposition ratings, anecdotal records, 
holistic scoring, and problem solution formats were used to edit the 
problems and amend procedures for the pilot. The need for time to meet 
more often and for inservicing was a primary concern in their 
recommendations. 

When asked about the impact on their teaching and student problem 
solving, all reported some positive effects of EQL Some of their comments 
follow. 

• / have begun to experiment with the introduction of problems 
before the required skills have been taught. Hopefully, this will 
give more meaning to the mathematical concept! 

• Vm integrating problem solving into many math tasks, not just 
treating it as a strand. It's incorporated into other subjects. My 
observation skills are improving. 

• [Students have] increased confidence. I had students score some of 
their own work holistically , and now they are more self-aware, 

• / believe my students are more confident about problem solving and 
have become greater risk-takers," They are very willing to try to 
solve problems and use more strategies than they did in September . 

Year 3: Pilot Testing 

The project activities in the third year centred around the pilot testing of 
materials in local classrooms and the gathering of results from that pilot test. 
This section reviews the following activities of the third year: 1) pilot testing 
activities; 2) analysis of pilot testing data; 3) development of 'Troblem 
Solving Profiles" handbook; and 4) findings and conclusions. 
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Pilot Test Activities 



The EQI-Math materials were piloted in eight Calgary Board of Education 
and Calgary Catholic Board of Education classrooms from September to 
December 1991 . Pilot teachers were identified by the system mathematics 
specialists as being successful, professionally involved teachers of 
mathematics. If school principals gave their consent, prospective pilot 
teachers were asked if they wished to volunteer. The pilot teachers had no 
previous involvement with the EQI-Math project. 

Selected pilot teachers attended two days of inservice for which they were 
given release time from their teaching duties. The EQI-Math advisors and 
a professor from the University of Calgary provided sessions covering all 
the aspects of the pilot process, from the philosophy of problem solving to 
ftUing out the necessary project forms. This section includes: a) problem- 
solving dispositions, b) preparing anecdotal records, c) presenting the 
problems, d) problem rating, e) documenting classroom conditions, and 
f) providing feedback about the effects of piloting. 

a. Problem-solving dispositions 

Pilot teachers observed their students at the start of the pilot, and 
recorded whether or not each of the five dispositions was "displayed" 
or "not displayed" or "not observed." The five dispositions are: 

• motivated 

• creative 

• confident 

• strategic approach 

• strategic process 

Pilot teachers recorded a second observation of student dispositions 
during the administration of the fifth problem in the pilot set. Record 
sheets provided to teachers did not contain the information about the 
first teacher obser^'ation. 

b. Preparing anecdotal records 

The pilot teachers practiced preparing anecdotal records during the 
inservice by viewing a videotape of student problem solving and 
recording observations. Three students were observed during each 
of the problem assignments and records made on structured anecdotal 
records forms. Teachers were asked to select one student they 
thought to be a good problem solver, one who is a weak problem 
solver, and an *Tm-not-sure-about-problem solver." A sample 
anecdotal record appears in Appendix C. 
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c. Presenting the problems 



Pilot teachers administered six problems to the selected pilot classes 
during the same weeks. Teachers could allow students to collaborate 
with their peers at the beginning of the problem session, but all 
students were required to submit a solution for rating, noting whether 
they collaborated with others on their solution sheet. The time 
allowed for each problem was determined by the teacher. 

Four graduate education students interviewed two students at each 
problem administration. The students were coached in "flexible 
interviewing techniques" and consulted with Dr. O. Chapman, a 
mathematics professor, as necessary. Both video and audiotapes 
were taken of each problem-solving session, and the audiotapes were 
transcribed. Excerpts from the transcripts are included in the 
"Problem Solving Profiles" to illuminate processes of student thinking 
that are not included in written responses. 

± Problem rating 

The Holistic Response Criteria were examined and discussed at the 
pilot teacher in service. Pilot teachers were encouraged to sort the 
student responses to the EQI problems into three groups: 
"Preliminary," "Partial" and "Complete." Following that, they were to 
review the "Complete" set, and select any responses that were 
"Elegant." 

Teachers rated their students' responses using the holistic response 
criteria, and recorded this rating on a form sent in to the project office. 
The problem responses were then sent to the colleague pilot teacher 
at the same grade level for a second rating. This pilot teacher then 
sent the second score and the student solutions to the project office. 

e. Documenting classroom conditions 

The pilot teachers responded to questions about their teaching 
practices and beliefs at the beginning of the pilot process and at its 
completion. These responses formed tiie basis of our analysis of 
changes in teacher attitudes and behaviours as a result of using the 
EQI-Math materials. A copy of this survey form, "EQI-Math Problem- 
Solving Practices and Beliefs" is included in Appendix C. 

EQI-Math advisors visited each pilot classroom and completed an 
"Environmental Scan." The scan attempted to describe the classroom 
environment through a series of responses to questions about the 
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classroom organization, teacher practice, materials available, and other 
resources. 

Providing feedback about the effects of piloting 

At the final pilot teacher meeting in January 1992, the pilot teachers 
provided both written and oral responses to questions of interest 
regarding the results of the project. All of the responses were 
maintained and provide a rich source of qualitative data for the 
preparation of project reports. 

Analysis of Pilot Test Data 

Problem-solving dispositions 

Disposition records for the first and second observations were 
compared. Data analysis showed that in the second observation, one 
or more dispositions were observed in 64% of students, an increase of 
16% over the first observation. The most substantial change was 
reported in the strategic process disposition. For all grade levels, 
correlation coefficients between dispositions and problem ratings as 
measured by Cramer's V were low to moderate, ranging from .06 to 
.38. 

Preparing anecdotal records 

The anecdotal records developed by the pilot teachers were reviewed 
by Olive Chapman, mathematics professor at the University of 
Calgary. The records have been used to develop materials regarding 
observation and flexible interviewing for classroom teacher use. 

In reviewing the anecdotal records kept by teachers, it was apparent 
that teachers used a variety of approaches, were not uniformly 
prepared to do the task of documentation, and found the processes of 
drawing inferences and teaching recommendations very challenging. 
Some of these factors might have been eliminated if more time had 
been spent inservicing and preparing teachers to observe and record 
student behaviour, and to make inferencf*s and teaching 
recommendations as a result. 

Student problem ratings, attitudes, and dispositions 

The EQI-Math project was designed as action research, and no 
research questions or proposals were established prior to the 
implementation of the project. However, some data were collected in 
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the process of completing the project activities and these data were 
available for whatever information they might yield. 

Documenting classroom conditions 

Prior to the pilot, the selected teachers were asked to respond in 
writing to a set of questions regarding their beliefs and practices. At 
the conclusion of piloting, teachers were asked to respond again to 
the same set of questions. The two sets of responses were analyzed 
to permit identification of changes in individual teacher's comments 
over time. 

The EQI-Math Advisory Committee reviewed the responses and 
identified changes in teacher behefs and practices. Responses to 
questions about problem-solving beliefs did not seem to change 
significantly between the two surveys. Teachers selected for piloting 
were already strong teachers, with furmly held beliefs about learning 
and teaching. However, the pilot teachers did report some changes in 
their practices and some plans for change in the future. Some of their 
comments follow. 

• The project has made me do more problem-solving lessons. It 
also helped me track students differently. It gave me more 
definite ways to categorize what I saw. 

• [I will] be more conscious of the process rather than the end 
product. Give students plenty of problems at the beginning of 
the year and not mark them ...no risk involved. 

• / now mark holistically because it gives a clearer picture of the 
strategies students are using, and I also have a better idea 
about which students would benefit from interviews. 

Providing feedback about the efTects of piloting 

Math pilot teachers responded in writing to the question: "Are your 
students better problem solvers as a result of your participation in the 
EQI-Math pilot?" 

Students were seen to be better problem solvers to some degree in 
every pilot classroom. Pilot teachers provided many comments 
suggesting that improvements occurred as a result of risk-taking, 
working in groups, encouragement to try different ways, increased 
discussion, and generally, a more positive attitude. Some of their 
comments follow. 
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• The encouragement of multiple solutions and creative responses 
allows children to be individuals, and respect their choices, 

• [There is a] strong component of talk and the need to explain, 
clarify, work through ... such that students developed confidence 
to use their ''voice, " 

• / believe they are better now because I'm looking at them 
through a new pair of eyes — this project has shown me a few 
new things about how students attack problems. Thus I can now 
better address the problems they may have and I think J can 
better appreciate why some are having difficulties ... hopefully, 
this will help us all to become better problem solvers! 

The pilot teachers also discussed their responses to the various 
aspects of the EQI-Matl-i pilot — the hoUstic scoring, disposition 
identification, anecdotal records, student attitude reports, and 
conditions that encourage quality problem solving. The summary 
notes from these discussions were prepared by the math advisors. 

Holistic scoring was well received by all teachers although secondary 
teachers were concerned with becoming more precise and expanding 
the descriptors. In general, the focus on the process over product, the 
lack of a discouraging "mark," and the sharing of expectations about 
a good solution were seen as very positive features. 

Disposition identification was positively received by most of the 
teachers because it "focused the observations" and provided a tool 
and vocabulary for reporting. At the junior high school level, there 
was concern that trying to make a judgement about dispositions from 
observation was sheer guesswork, and there was great variety in 
students from one problem to the next. 

Anecdotal records were criticized by all pilot teachers as being 
"frustrating," "time consuming/' "inaccurate," or "interfering with the 
teaching process/' A few positive comments related to the possible 
use in student interviews. 

Student attitude reports were criticized by pilot teachers at all grade 
levels. In the early grades, teachers felt that the students didn't really 
understand the words. In the senior grades, there was concern about 
the vocabulary used, and whether students took the response 
seriously. A few students at each grade level were seen to give the 
response a lot of thought. 
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Conditions that encourage quality problem solving were similar for all 
grade levels. Teachers identified time, the opportunity to work in 
small groups, the free discussion of ideas without judgement, a 
facilitating teacher, resources in the classroom, and good problems as 
important conditions. 

The EQI-Math teacher researchers and pilot teachers provided a large 
amount of qualitative information about their experiences and 
observations throughout their work with the project. This 
information was reviewed by an evaluation specialist who prepared a 
summary questionnaire based on the information in the qualitative 
responses. This questionnaire and the results are included in 
Appendix C. 

Nineteen (79%) of the 24 teachers who had been involved with the 
project responded to the survey. All of the respondents agreed that 
their participation in EQI has been a valuable learning experience, 
and 95% agreed that involvement in EQI has made them a better 
teacher of mathematics. Teachers at all grades who participated in 
the EQI project experienced the greatest increase in skill and 
confidence in the area of assessment of student growth in problem 
solving. 

During the piloting of EQI-Math materials, the mathematics advisory 
committee visited all pilot classrooms at least once, and prepared an 
"environmental scan" regarding the classroom conditions. 

Math advisors expressed discomfort with using one or two limited 
observations as indicative of a classroom environment, so this 
information is not used in the project findings. 

The two pilot teachers at the grade 5 level asked students to respond 
in writing to their experiences during piloting. Students were given 
some class time and encouraged to contribute their ideas. 

The students reported that they learned about approaches to 
problems as well as interpersonal relationships in doing the EQI 
problems. The student responses all had positive elements. Here is 
one sample: 

• / really liked doing the problem solving. It was very interesting. 
I think it was a good idea to work in groups because if you 
didn't understand the problem, the group could help you, I 
learned that sometimes I chose the wrong strategies and I 
learned that I should check the problem. Sometimes my group 
did not make a plan and we got the problem wrong. 
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Development of "Problem Solving Profiles" Handbook 

The teacher researchers and the pilot teachers reported that discussing 
student responses with other teachers was an important benefit of 
participation in this project. Recognizing the value of this sharing of ideas, 
the EQI-Math advisors began in 1991 to plan the development of a 
"Problem Solving Profiles" book that would provide teachers with 
approaches to problem-solving assessment from the EQI project. 

The profiles contain information about student dispositions, holistic 
scoring, and flexible interviewing. For each grade level, a selection of 
student solutions illustrate samples of student work that show the 
characteristics of the holistic response criteria. Student solutions were 
selected from the pilot project. Each solution is accompanied by 
observations of expert math teachers who have been involved with the 
EQI-Math project. 

The "Problem Solving Profiles" handbook contains the elements described 
above. The handbook presents classroom assessment of three elements: 
1 ) what students do when solving problems; 2) how students think when 
solving problems; and 3) what students write down in response to 
problems. "What students do" can be observed and recorded in anecdotal 
records and linked to the student dispositions. "How students think" can 
be approached through the use of flexible interviewing techniques. "What 
students write" is shown in their problem solutions and can be scored 
holistically. The handbook provides a small amount of theory and 
practical examples that model assessment of each element of student 
problem-solving behaviours. 

Findings and Conclusions 

The Calgary EQI-Math Project identified indicators of quality student 
work in problem solving in mathematics, grades 2 to 10. Indicators can 
inform the educational practice of teachers and students in several ways. 

• Teachers can use EQI-Math materials to: plan and implement 
instructional strategies and programs appropriate to student needs; 
maintain accurate, informative documentation regarding the 
educational attainments of students; and report back to students, 
parents, administrators and others about these attainments. 

• Students can compare their efforts with models or exemplars of 
quality work, develop goals and personal standards to attain, and 
apply their energy to improving their learning. 
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After the development and testing of classroom materials designed to 
develop indicators of student problem solving, project advisors reported 
the following conclusions. 



• Teachers reported positive changes to the?r teaching and assessment 
practices as a result of participating in EQI-Math. 

• Teachers wanted to meet to discuss teaching and assessment. The 
experience of working closely with colleagues was very positively 
received. 

• Some of the materials and practices developed in the project were 
seen to be useful. Specifically: 

student dispositions were helpful in observing students, and in 
developing understandings about student thinking 

hohstic scoring was helpful in two ways: teachers looked 
beyond the answer to the processes used by students; and 
observation and reflection were encouraged 

looking at many student solutions helped both teachers and 
students understand and communicate about the diversity of 
responses and what quality problem solutions look like 

• Some of the materials and practices developed in the project were not 
seen to be useful Specifically: 

anecdotal records were very time consuming and inconsistent for 
use in large-scale problem-solving assessments 

student attitude reports were of limited value to teachers 

generalizations about classroom conditions that promote problem 
solving could not be provided from this project 

• Students seemed to be more positive about problem solving, took 
more risks, and demonstrated more confidence as a result of the EQI 
project. 

• Systematic methods for observing students closely — for example, 
the disposition sheets and extensive answers — helped teachers with 
teaching decisions and with assessment and evaluation tasks, 

• Students responded to the invitation to express themselves more fully 
and to think about their thinking. 
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The project activities did not determine to what degree there was an 
improvement of quality in student work as a result of the use of EQI 
matcTials. A further study would be needed to understand more about this 
relatioiiship. 

In conclusion, the project developed some promising approaches to the 
task of assessing student problem solving. It is a good start, and a basis for 
people to build from in developing performance assessment measures. The 
methods of holistic scoring and identifying dispositions contribute to 
broader measures of assessment that are useful in understanding and 
reporting on student achievement. 
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CHAPTER 5: SUMMARY AND DISCUSSION 



Action research begins without an end product in mind. The Calgary EQI 
Project began in just that "open-ended" way. Along the way, the 
researchers struggled with choosing from a variety of alternatives, 
answered many questions about assessment practice, and proposed other 
questions that remain to be answered. This chapter contains 1) findings of 
the research, 2) processes promoting success, 3) response to the project 
products, 4) future directions, and 5) recommendations. 

Findings 

Art and mathematics appear to represent very different assessment 
challenges, and we did not expect to find many commonalities in teacher 
practice. However, as the EQI- Art and EQI-Math projects evolved, many 
commonalities emerged as themes and styles of practice. The 
commonalities emerged because the making of art and the solving of 
mathematical problems require complex thinking processes, and their 
assessment requires informed professional judgement. The expression and 
development of professional judgement is the common focus of the project 
conclusions. 

Findings that are common to both EQI-Art and EQI-Math are: 

• Quality assessment models in art and mathematics support teacher 
decision making about teaching and assessment. The EQI materials 
were clearly intended for use in assessment practice: as the project 
unfolded, it became obvious that there was considerable positive 
impact on teaching practice. This finding supports the view of 
learning adopted by the project — students create knowledge. The 
challenge for teachers, then is to become skilled in listening and in 
observing student work and students at work. 

• Attending to written and/or spoken student reflections illuminates 
the art work or the problem solution. Both art and mathematics 
teachers found that the additional written and verbal explanations 
provided by students were a new "window to the mind" of the 
learners. In many instances, teachers found that assessing student 
outcomes without involving students to explain their thinking was 
very difficult. Teachers reponed that their attitudes and behaviours 
were changing as a result of "listening" more to students. Advisors 
suggested that more attention to the student "voice" would be an 
obvious further development of EQI materials. The importance of 
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language in student art making and problem solving was continually 
emphasized in the project. 



• Quality assessment is a shared experience between teachers and 
students, encouraged by shared exemplars and common 
vocabulary. Teachers reported surprise and delight at the interest 
and sophistication displayed when students became involved in their 
own and peer assessment. Teachers and students require 
opportunities for joining in discussion and examination of their 
intended meanings and problem solutions. 

• Qualitative assessment takes time since students must complete a 
complex assignment and teachers must observe and respond. The 
concem with not having time to do assessment of this nature was 
expressed more frequently by secondary mathematics teachers and 
was related to a perceived pressure to cover a required amount of 
prescribed curriculum. 

• Exemplars in art and mathematics performance assessment are a 
way to share expertise and to develop a sense of connoisseur ship. 
A collection of quahty exemplars in subject areas helps develop high 
and common expectations for student outcomes and celebrates 
diversity in student responses. 

Informed professional judgement is the critical component in the 
qualitative assessment of complex student outcomes in art and 
mathematics. The Calgary EQI Project used the informed professional 
judgement of expert teachers and subject specialists to develop and refine 
this project. The materials and approaches that have resulted model the 
practice of assessment for other teachers and for students. 



The involvement of classroom teachers in all phases contributed to the 
strength and validity of the project. Teachers contributed expert 
judgement, thoughtful reflections about teaching and learning, and precise 
criticism regarding materials and methods. Teachers showed how much 
they want to work with colleagues on issues of assessment and they 
reported receiving professional benefits and personal satisfaction from 
coUegial work in this project. 

The involvement and commitment of two large school boards provided a 
wealth of background not only in subject expertise but also in other areas 
of learning, teaching and assessment. Project teachers valued the 
opportunity to work with new colleagues. Project advisors all commented 
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on the benefits of working with colleagues in similar positions with 
another school board. 

Response to the Project Products 

Teachers and students have responded positively to the materials and 
approaches developed in EQI-Art and EQI-Math. Chapters 3 and 4 
describe in some detail the positive responses of teachers to the activities 
and products developed within the EQI project. There is some evidence 
that positive changes occurred in learning and teaching as a result of using 
EQI materials. 

Presentations regarding tlie EQI project have been very well received. 
Teacher and administrator groups want copies of the art handbook and 
mathematics materials to work with in their development of assessment 
measures. Business people support the focus on significant outcomes and 
quality work. Parents comment on hovv' the exemplars help them 
und rstand expectations. University personnel want to share these 
approaches with teachers in training. 

Future Directions 

Teachers and students have responded positively to the materials and 
approaches developed in EQI- Art and EQI-Math. There is some evidence 
that positive changes occurred in learning and teaching as a result of using 
EQI materials. 

The development of the EQI assessment materials required time: to discuss, 
develop, experiment, rethink, assess, evaluate, and to begin again with new 
ideas that emerge. We were provided with support for three years, and 
made promising advances. To continue this developmental process 
requires that more time be dedicated in the future. 

Recommendations 

At the conclusion of the Calgary EQI Project participants at all levels 
assisted in the preparation of recommendations for future development of 
the project ideas and materials. The participants recommended that: 

• new assessment processes be implemented into evaluation and 
reporting practices; 

the EQI-Art Handbook and the Problem Solving Profiles be 
published and made available to schools within the CBE, the CCBE, 
and to other school systems on request; 
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• institutions involved in teacher preparation include a variety of 
assessment skills in teacher-preparation courses; 

• teachers be encouraged to develop professional collections of quality 
student exemplars in art and in mathematics; 

• teachers be provided with opportunities to work on collegial 
assessment of student work; and 

• school jurisdictions be invited to expand the EQI project process to 
include other disciplines. 

In conclusion, this project reaffirms the importance of thoughtful reflection 
on the part of the students and teachers in the assessment of student work. 
It is also clear that innovation requires system support, guidance and time. 
When teachers and students are informed by experts, connoisseurs and the 
literature, they have the knowledge and skill to engage in meaningful 
assessment. 
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APPENDIX B 
EQI-ART MATERIALS AND REPORTS 



Handbook Example 

• Sample Lesson from EQI Art Handbook 

• One Student Sample from the Lesson 

• "Quality Indicators in Art" 

Review of Teacher and Student Journals 
EQI-Art Pilot Teacher Survey 
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Lesson 1 



Grade 2 



Directed Observations and Design: Water Motion 



The teacher challenged students to reflect that environments are altered by natural forces and to 
observe that shapes can suggest movement. Students v/ere to observe examples of the changing 
patterns created by animals moving through water and to note how light is reflected when water is 
moving. Using oil pastels, students were challenged to depict movement through water. Students 
observed many examples from art and from the teacher's picture file. Students had a choice of 
background and of animal. 

The teadier expected that the students would: 

• imagine an animal swimming through water 

• create a two-colour oil pastel drawing showing how water changes when an animal swims 
through 

• make patters that show light and shadow on the surface of the water 

• indicate the movement of the water's wake 

Materials: coloured construction paper 
Medium: oil pastels 

Time: two 45-minute class periods in the fall term 



Student Reflections: 

/ drew a dog swimming in the water. Whatever 
is in the water is making a wal^e coming 
behind it The waf<e makes iines and patterns 
in the water. / wanted blues so that it wouid 
go good and brown to coiour the dark. 




Observations: 

The student has created an unusual pattern of blues, browns and white to show water movement. 
Shapes have much variety in design, size and placement. The sharp contrast between the pattern 
In the moving water and the absence of pattern in the calm water creates a dramatic feeling of an 
animal forcing its way through water. 

Repeating diagonal lines, colours and shapes makes unity. The dark shape of the dog placed off- 
centre creates emphasis, movement and contrast. 

The handling of the oil pastel seems confident. Original work is 12*^ x 18" 
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QUALITY INDICATORS IN ART 



Meaning 

To what degree does the work successfully EXPRESS THE IDEA OR FEELING the student intended? 



Relationships 

• UNITY: The essential characteristic: everything fits. 

How do all the parts of the work - shapes^ lines, coiours, textures, values • relate to one another 
and to the main idea? Consider questbns of balance, of warm and cool colours, etc. 

• PATTERN: rhythms made by repeating elements. 

What lines, shapes, colours are repeated? 

What variations in repeated elements are noted? 

How do the repeated slements strengthen the idea/mood of the worl<? 

• EMPHASIS: a focus in the work. 

What shapes, lines, colours are emphasized? 

How does the use of emphasis support the idea/mood of the work? 

• CONTRAST: shapes show up. 

How does the student use dark and light, hard or soft edges or textures to communicate the 
kJea/mood? 

• ELABORATION: significant details. 

Where have details been added? 

How do these details strengthen the idea/mood? 

» MOVEMENT/SPACE: lines, colours, shapes, space movements create paths for the eye to follow. 

Is the space deep or shallow? 
How are positive and negative space handled? 

Handling 

To what degree does the work show confidence, control and flexibility in the use of the materials? 

• CONTROL OF THE TECHNIQUE: student has mastered the needed skills. 

• FREEDOM AND SECURITY: use of medium • paint, chalk, graphics, etc. - shows confkJent handling. 

• THOUGHTFULNESS: work shows evidence of care and planning. 

• EXPERIMENTATION: work shows new uses or several uses of a chosen medium. 

- ABILITY TO USE THE EXPRESSIVE CHARACTERISTIC QUALITIES OF THE MEDIUM: softness of 
chalk, fluidity of paint, sharpness of pen and ink. etc., support this idoa. 
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Educational Quality Indicators Project 



REVIEW OF TEACHER AND STUDENT JOURNALS 

Submitted to Jane Sereda 
Aprils, 1992 

Michelle Zlederayko 
Margaret Latta 

Considerations 

The consideration of the EQI journals provides some insights into the nature of 
the project on one hand and the difficulties of getting art students to write about 
their art on the other hand. Use of the journals ranged from abstract ramblings 
which were very student-diiected to a set series of questions which were 
provided by the teacher and attached to the journal. 

At the outset, it was interesting to no*e that each classroom was notably different 
in approach and that the student comments appeared to reflect the individual 
classroom. Student responses ranged from the student who suggested that, "it is 
quality work because I have put a full four classes into the project," to the student 
who asserted that, "it is quality work because the shoe looks like the shoe I was 
supposed to draw." One obscure reference to the quality Indicators Handbook 
suggested that the student's work was quality work, "because I compared them 
and they were dmcst the same." It is assumed that the comparison was between 
the student work and the work in the Handbook, although this is not made clear 
in the journal. 

The suggestion that being"almost the same" meant that quality had been 
produced raises some concerns about the role of the Handbook in the classroom. 
A student producing facsimile works oi art because he/she believes it is a sure-fire 
way to produce quality is a frightening prospect and one which merits further 
attention. It suggests that, in order for the Handbook to serve a useful function in 
the classroom, its use needs to be highly mediated by a teacher who is 
himrelf/herself comfortable with the idea. Classrooms where the teacher journal 
suggested a high degree of anxiety (ostensibly due to the fact that the teacher 
did not feel that he/she was enough of an "expert") also resulted in student 
journals which tended to be very lockstep in their consideration of the notion of 
quality in their art. Classrooms where the teacher's and student's journals 
suggested a high degree of reflection resulted in student comments which did not 
appear to consider the Handbook. One such student, who was discussing the 
degree of completion in his/her picture said, "When I look at a picture I ask myself 
is it really done, and to an jwer that I compare the image in my head to my 
picture." 
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It is hard to accurately assess the merits of the program in the ligh t of journals 
from students in tl.^e lower elementary grades since the task of writing became a 
real impediment to an accurate recording of the artistic process. Since the 
concept of reflective practice in the art making process is inherently good, this 
problem could possibly be circumvented through use of a tape recorder. We feel 
that the value of dialogue as a means to sense-making cannot be overstated. 

Perhaps the most significant implication from a study of the journals was that 
even at the elementary levels, there was a notable difference in both stud<°int and 
teacher response between classes where the teacher appeared knowledgeable 
and classes where the teacher appeared to be doing his/her best with limited art 
training. Teachers with art backgrounds provided their students with a more 
extensive vocabulary of art terms and, to judge by the kinds of comments 
appearing in their students' journals, a concurrent understanding of basic art 
fundamentals. The students in classes where the teacher's journal reflected a 
strong art background showed a far greater inclination to use their journals as 
workbooks which contained plans for future works, as well as thumbnail 
sketches and insightful discussions of past works. 

Conversely, student journals from classrooms in which the teacher's journal 
entries suggested a lack of comfort with the subject area produced substantially 
different journals. Many of these students wrote very little in their journals and 
what they did vmic appeared to be very teacher-directed, albeit with little use of 
an extensive art vocabulary. That is, these students appeared to write in their 
journals primarily in direct response to questions posed by the teacher and their 
responses often did not reflect the same degree of informed thought about the 
nature of the artistic process. 

Thus, the journals would appear to suggest that the EQI Handbook becomes a 
viable teaching tool in the hands of a teacher who has a fairly high degree of 
comfort in teaching art. The journals would suggest that a teacher with little art 
experience is inclined to be either threatened by the Handbook, or to use the 
Handbook as a type of Art Recipe book. 

Teacher comments indicated a desire for more extensive inservicing and support 
for using the Handbook, and several pointedly suggested that the actual "how 
to" of using the Handbook effectively in their classes was a difficult practical 
concem. One teacher, who saw the Handbook as a selection of "model lessons," 
pointed out that more information on teacher motives should accompany each 
project in the Handbook. Another teacher raised a cautionary note in regard to 
the use of the Handbook as the start of a project since it "stifled the creative 
process because they (the students) think that it is the only moGc^l." 

Clearly, the EQI Handbook has some exciting, as well as sobering, implications for 
its use in the classroom. The implication which requires the most thought, 
however, would appear to be the fact that no handbook, no matter how good or 
well-thought-out, can replace expertise on the part of the teacher. 
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Implications 



The Art Making Experience 

During art making or creating, many students wrote in their journals of ^eing 
involved with the medium. They described art making as a "thinking, 'working," 
"constructing," "changing," "building" process. Seemingly, the art-making 
experience required a relationship between the artist and the medium. It was a 
search for qualities - the perception, selection, and organization of qualities and a 
responsiveness to them. The use of the journals by art teachers and art students 
held the possibility for engaging them in thinking, feeling, reflecting, and talking 
throughout the art-making process. The facilitation of this dialogical sensi^- 
making process was the strength of the journal use by teachers and students. 

Some journals evidenced progressively much more depth and insight than others. 
Acknowledging that there were many factors that could have contributed to this, 
we felt generally that students who had been encouraged to involve themselves 
in the art-making process, valuing this process over the product, and where an 
ongoing dialogue was promoted, evidenced greater understanding in their 
journals. We surmised that the dialogic sense-making process was dependent on 
involvement, dialogue and community. These are intertwined and dependent on 
each other as the art form was created. Above all, the teacher's place in making 
this happen, was central. 

Involvement: Inquiry Guided 

Individual interpretation should be encourage. Art making is a search that starts 
with the self. Students that were encouraged to "play" with a medium 
discovering for themselves the nature of the medium and a solution of their own 
making, made comments in their journals that reflected greater sense making and a 
tacit knowing developing . . . "slowly learning more and more about drawing"; "I 
decided not to add another colour there because I felt it would be a distraction"; 
"I said no to myself because I decided it v/ould be too much." These same 
students seemed far less concerned with the end product and more interested in 
critically reflecting on their own work and others' in progress. 

Dialogue and the Place of Community: A Discourse by Nature 

A large part of the search process is the ongoing dialogue generated between the 
emerging art work and the artist. The attention drawn by teachers to quality 
indicators seemed to encourage students to see the interrelationships of elements 
with in a whole. It also helped students find the words to verbalize their 
reflections on their own and their peers' art work. This discourse was greatly 
facilitated by teachers who responded to their students' work in a manner that 
continued the conversation rather than shutting it down. 

8S 
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In Conclusion 



We saw the journals as holding the possibility for being an excellent means to 
foster artistic learning. The opportunity for teachers to reflect on their own 
practices was described by many as "invaluable," Their constant questioning and 
striving to understand the project was made clearer through their reflections, 
observations and feelings about their art classes and students. We saw the place 
of the teacher as pivotal in regards to facilitating the use of the journals in an 
approach that promoted a continuing and evolving discourse. The students* 
journals fostered reflection on their art learning, a heightened sensitivity to 
elements within a whole developed. It is hoped, therefore, that this encouraged 
quality improvement in student art work. 
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EQI-ART PILOT TEACHER SURVEY 



PURPOSE OF THE SURVEY: This survey is designed to gather information about the art 
assessment beliefs, practices and concerns of teachers in the CBE and CCSB in order to 
develop assessment programs and materials for Art teachers. No individual responses will be 
used in reporting, and you need not identify yourself on the questionnaire. 

If you have questions or comments about the survey, please contact Jane Sereda, Educational 
Quality Indicators Project, Calgary Board of Education, telephone 294-8243. 

•Id but of 11 pilot teaictiers (91^) r^]k>nclB<i Vo'the survey 

1. 1990-91 ART TEACHING ASSIGNMENT 



Division I (Grades 1-3) Average 

Division 11 (Grades 4-6) Average 

Division III (Grades 7-9) Average 

Division IV (Grades 10-12) Average 



1.5 hours per week Art instruction 

6.5 hours per week Art instruction 

8.0 hours per week Art instruction 

15.0 hours per week Art instruction 



2. NUMBER OF YEARS TEACHING EXPERENCE - Write the number of years you 
have taught in each Division on the line below. The mixed Division category is for 
teachers who have had teaching assignments across Divisions, 



E.C.S Div.I 



Div.I/II 



Div.II DivJVIII DivJII DivJII/IV Div.IV 



TOTAL YEARS ''TEACHING" 



NUMBER OF YEARS OF EXPERIENCE TEACHING ART - Write the number of ytais 
you have taught Art in each Division on the line below. 

E.C.S Div.I Div.I/II Div.II DivJI/III DivJII Div.HI/IV Div.IV 



TOTAL YEARS "TEACHING ART* 
"Xm^e nuiubcr.of years Hekching zrV^^yt^^ tiJ-y^rsJ' 



4, mJMBER OF UNIVERSITY OR COLLEGE ART COURSES TAKEN (one full course 
and one half course would be written as "1.5" courses) 

Art Methods Average 2.4 courses 

Art History Average 2.4 courses 

Art Studio Average 8.0 courses 
other Art Courses (specify): 
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5. In the past two school years (since September, 1989) about how many hours have you 
spent directing in art inservices and/or giving professional assistance related to the 
teaching or the assessment of art? 

Average 21.2 hours 

Range from 10 - 50 hours 



For each of the following descriptors, please circle the number that represents the 
degree to which each describes YOU: 

very descriptive not descriptive 

a practising artist 1 2 3 4 5 

a specialist teacher (in Art) n ^'S . ' 1 • } 2 3 4 5 

a non-specialist teacher (in Art) n ^'^Z^^y^^M^ 3 4 5 



7. If you had your choice of teaching assignments, would you: (Circle the ONE number 
that best represents your choice.) 

a. express a preference to teach Art ov^i^VS^^thp^^ ' 

b . express a willingness to teach Art - i out of 10 rdspo'nd^nts 

c. express a preference not to teach Art 

If you answer "c," please explain why: 



Beliefs about Art instruction and assessment are illustrated in the statements that follow. 
Please read and respond to each of the following statements by circling the letters that 
indicate your level of agreement with each statement on the scale: 

All responses from this point on reflect "after pilot" results. 



SA = Strongly Agree 

A = Agree 

U = Undecided 

D - Disagree 

SD = Strongly Disagree 

I 

Art instruction has a significant effect on a studciit*s 
ability to produce good art work. 
It is important for all students to have quality Art 
instruction as part of their public school education. 
Innate talent determines how well students perform 
in An. 

The identification of quality work in Art is based on 
individual taste. 



SA 



2 
1 

6 
7 



U D 



SD 
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SA 



U 



D SD 



e. All students are capable of developing good skills 
in Art depiction and composition. 

f. Art courses should be given the same level of 
importance as all other courses in the school 
curriculum. 

g. Objective criteria should be the basis for assessing 
quality work in Art. 

h. It would be helpful for teachers to have common 
criteria for assessing the quality of student art work. 

i. Art cx)urses should be provided with a similar 
amount of resource support as are other courses in 
the school. 

j. Checklist are more valid than individual critiques in 
assessing student art. 



:1'facre;wai^^^^^^ in these 'answers -be^^ and after/pifo 



9. The statements below represent impressions of the status of Art in a school. Please 
indicate your level of agreement about how well each statement is representative of the 
general feelings IN YOUR SCHOOL on the scale: 

SA = Strongly Agree 
A = Agree 
U = Undecided 
D = Disagree 

SD = Strongly Disagree —r- 







SA 


A 


U 


a. 


Art is an important subject. 


4 


3 


3 


b. 


All students should take Art from Grades 1 to 6. 


3 


5 


1 


c. 


All student should take Art at least once in 


4 


3 


1 




secondary grades. 








d. 


Art is a source of pride for the school. 


5 


4 




e. 


The objectives of the Art program are understood. 


3 


3 


3 


f. 


The Art program turns students on. 


5 


5 




g- 


Art is highly visible in the school. 


7 


1 


1 


h. 


The Art program receives its share of resources and 


4 








support. 








i. 


At our staff meetings, Art receives the same 


1 


3 


2 




attention as other school subjects. 









2 



3 1 
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10. Below is a list of assessment methods some teachers use in Art programs. For each of 
the methods listed, please indicate how often you use each method in your current Art 
classes by CIRCLING the appropriate term on the right. 

Except ivNre note^, Qtere was aimo$t m ^ange ill re$|KM|i$es W^rk aiid 0ier; 



ALMOST 


QUITE A 


SOME- 




ALMOST 


ALWAYS 


LOT 


TIMES 


SELDOM 


NEVER 



written tests 

less after piloting 

checklists 

portfolios 

visual journals or sketchbooks 

written critiques of individual art 
works 

verbal critiques of student work 
student self-ratings 

slight increase after piloting 

peer ratings 

slight increase after piloting 

numeric scores on art handed in 

observations of work habits 

other methods I use in assessing 
Art 



1 
6 
1 
1 



1 

3 
3 
2 

6 
2 



4 
1 
4 
5 



9 
2 



2 
1 

1 

2 



1 

4 

2 
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1 1 . Below are various categories teachers use in marking student achievement in Art 
courses. Beside EACH OF THE CATEGORIES, indicate the percentage that category 
contributes to the report card mark you give students. If you DO NOT use the 
category, please put in a ZERO. The sum of all entries should be 100%. 





Range 


Mode 


student knowledge of art theory 


0- 


50 % 


0 


student artwork produced 


16 - 


50 % 


20, 50 


attitude to the subject 


2 - 


20 % 


10 


attendance 


0- 


12 % 


0 


willingness, cooperation 


0 - 


20 % 


10 


group woric 


0 - 


13 % 


0 


creativity/originality 


1 - 


30 % 


20 


completion of project on time 


0 - 


15 % 


0 


other factors (specify): 
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12, Below is a list of factors that may play a role in how you plan your Art lessons. Circle 
the number that indicates the level of importance each factor has on your practice. 



a 

B 

I 



I 



I 
I 



I 



E 



integration with other subjects 


4 


2 


1 


3 




the Art curriculum guide 


3 


3 


4 






the Art Resource Guides 


3 


3 


3 






recommendations from school 
administrators 




2 


3 


4 


1 


personal collection of successful lessons 


3 


5 


2 






perceived student needs 


5 


4 








recommendations from colleagues 




3 


4 


2 


1 


recommendations from system art 
specialists 


4 


2 




3 




cultural celebrations in the school 




3 


1 


5 




cultural celebrations in the community 




2 


1 


5 


1 


seasonal art (leaf art in fall) 




2 


2 


4 


1 


my own judgement as a teacher 


6 


3 


1 







Other: (specify) 
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14. What are the limitations you see in delivering a quality art program in your school? 



Thank you for your cooperation. 
September, 1991 
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APPENDIX C 
EQI-MATH MATERIALS AND REPORTS 



Year One: 

EQI-Math Year One: 
Year Two: 
Year Two: 
Year Three: 
Year Three: 
Year Three: 
Year Three: 



Mathen.. ^al Dispositions 
Sample Problems 
Disposition Checklist 
Inventory for Sample Student 
Mathematical Dispositions 
Holistic Response Criteria 
Sample Anecdotal Record Form 
Problem-Solving Practices and Beliefs 



Survey of Teacher Participants in the EQI-Math Project 
Math Problems used in Piloting 
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EQI-Math Year One 



Mathematical Dispositions and Observable Cliaracteristics 



Perseverance 

reluctant to stop, willing to persevere 
in mathematical task** 
tries different methods* 
not easily discouraged 
works to a goal 

completes work (even when not 
assigned for homework) 
works for a lengthy period of time on 
one strategy 

Risk-Taking 

not afraid to make mistakes (several 
trials and errors) 

flexible in exploring mathematical 
Ideas and trying alternative methods* 
asks questions 

positive attitude toward trying new 
things 

Confidenca 

confident in using mathematics to 
solve problems, to communicate 
ideas, and to reason* 
confident that an answer will be found 
sure of correctness even when 
explanation Is difficult 

Motivated 

actively doing something (asking, 
charting, computing) 
shows interest, curiosity and 
inventiveness In doing mathematics* 



Achieving 

sets goals 

successful in many areas of schooling 
values recognition for 
accomplishments 



Shows Understanding 

knows the meaning of the words in 
the question 

can state the problem in their own 
words or picture or model 
knows what the question asks for 
has some idea of the form the answer 
will take 

recognizes a reasonable answer and 
discards an unreasonable answer 

Communicativa 

provides verbal or written explanation 
of response 

uses mathematical language* 
talks through problems with others 
shares ideas, piggybacks on the ideas 
of others 

Craativa 

own problems, formulates problems* 
enjoys finding more than one solution 

Strategic/Organized 

looks at problems from different 
directions 

uses logical approaches, classifies, 
estimates 

sorts data to see patterns 
applies mathematics to situations 
arising In other disciplines and 
everyday experiences* 

Reflective 

inclined to monitor and reflect on their 

own thinking* 

learns from mistakes 

appreciates the role of math in culture, 

as a tool and as language* 



* Indicates terminology used in Curriculum and Evaluation Standards for School 
Mathematics, NCTM (1989) 
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EQI-MATH YEAR ONE: SAMPLE PROBLEMS 



EQI-2 Teacher # 

Problem #1 Name 

Date 

SHOW AND EXPLAIN ALL YOUR WORK 

Mary Jane saw 5 rabbits. 

How many ears did those rabbits have in all? 



Maik an X on the face ±at shows how you feel about this problem. 
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EQI.5 



Teacher # 
Name 



Problem #1 



Date 

SHOW AND EXPLAIN ALL YOUR WORK 



The sum of my digits is 2. 
I'm greater than 12. 
Who am I? 

PlajinmgBoani (Whatdo wetaww? WhudowewanttDfindout?) 



Doins/Dnwing Board (Solotion, Dugram, Qun, Giid) 



How I feel about this problem 



Explain 



1 04 
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EQI-S 

Problem #1 



Teacher # 

Name 

Daw 



SHOW AND EXPLAIN ALL YOUR WORK 

Margie is i blonde. Rose Mary is a redhead, and Shiiiey is a bruxtene. They aie mamed to Alex. 
Frank and John, but 

(a) Shiiiey does not like John 

(b) Rose Mary is manied to John's brother 

(c) Alex is matiied to Rose Mary's sister .:> 
Who is oiamed to whom? (Assuiae that manied people like each other.) 







llQdcrsund 














._ ^ 






Soivc 


o 




Look Back 

















Qxcie words thai idl bow you fed about this problemorwhie your own words. 



Easy ChallengiAg Gsnfusing Fud. or 
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EQI-10 

Problem #1 



Teacher # 

Name 

Dare 



SHOW AND EXPLAIN ALL YOUR WORK 



1 X 1 paving susncs are used to make a bonier around a rectangular pond. In the examnle below 
the pond is 3 x 9 and 28 stones are used. 





1 1 






Pond 






__ hi: h 





What arc the different dimensions of {)oads dux would ttquire 36 ptving stones to complete the 
border? 



Grcle words duu tell how you feel about diis problem or write your own words 
Boring Challenging Confusing Fun or 



1 f 
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EQI-MATH YEAR TWO: DISPOSITION CHECKLIST 



































Student Name 


































































































































































































































































































































































































































































































































































































































































































































































































































































































































































































































































































































































































































































































1 

































Confidence 
Perseverance 

Riek Taking 
Understanding 



AchievcKient 
Orientation 



Motivation 

Communication 

Creativity 

Organization 
Reflection 
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EQI-MATH YEAR TWO: INVENTORY FOR SAMPLE STUDENT 



What student has 

demonstrated: Grade 2 Date and Activitv 

#13 Nov.- Dec./90 



1. 


CoD£dence in using 
matbematics 


- law risk with new ideas/concepts 
' confidau on r tasks 

• no corqidence in problem'Sobrng taskSf always seems 
unsure who: to do/how to proceed 


2. 


Perseverance at 


- not at probhm^sohnng tasks • sits and looks around 

needs to be prompted 
' low perseverance le\Hd - shows more avoidance 


3. 


Risk-talring in doing 
znatbemancai tasks 


" prefers safe, known type of tasks 

- no confidence in new ideas - needs to be prompted 

- won*t even really try - prefers to copy paroier or tries to 
avoid doing the task 


4. 


Understanding aix)iu 
matfaemancs 


* appears to hove a good basic understandaig (noted 
during calendar dms)^ yet differmces arise in new 
stuaaarts 

• lack of cortfidence t^eas perfonrumcr. 


c 
J. 


Achievement orientation 
towards matfaexnancs 




0. • 


iviouvauon lo go 
mailiemaiics 


- enjoys iriaxn 

* docs tasks willingfy - centers 

- gives impression of liking Math 


/. 


Comnmniration about 
maiiiftTnarics 


- atsie to yeruaaze a taste ne taiows^ out lauauy snyf 
unwilling to discuss 

- when he is tmsure or doesn't know , shrugs shoulders 
7 don't know"* 


8. 


Creativity in doing 
matiiemancs 


- not ob\ious/not creative 

• too busy trying to avoid the task 


9. 


Organizadon in doing 
mathemaDcs 


- organized when he understands the problem 
' ndies on others to give duo what to do thau^ 


10. 


Reflective about 
mathemarical rhnilriTig 


- not really r^lecdve 

- doesn't appear to reflect on what he is doing or what 
he needs to do in a task 
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EQI-MATH YEAR THREE: MATHEMATICAL DISPOSITIONS 



• Motivated 

shows interest, curiosity, eu usiasm and inventiveness in doing 
mathematics* 

goes beyond the assignment 
reluctant to stop* 
works to a goal 

works until satisfied with solution 

works for a lengthy period using one strategy 



• Creative 

flexible in exploring mathematical ideas* 

tries alternative methods* 

invents own problems, formulates problems 

nble to improvise in selecting materials to represent the problem 



• Confident 

belief in ability to use mathematics to solve problems* 
belief in ability to reason* 
not afraid to make mistakes 
- belief that problems can be solved 

ability to identify and use necessary resources 
prepared to demonstrate for other students 
belief in ability to communicate ideas 



• Strategic Approach 

can state the problem in own words 
understands the meaning of words in the problem 
is able to draw a diagram that reflects the problem 
- knows what information is needed to solve the problem and what 
information is extraneous** 



• Strategic Process 

uses organized and logical approaches 
takes time to stop and think 
notices and corrects one*s mistakes** 
monitors one's thinking* - is reflective 

* Terminology used in NCTM, Curriculum and Evaluation Standards for School Mathematics, 1989, p, 
228 

** Terminology used in Alberta Education, Diagnostic Mathematics Program, 199 1 , "Cognitive 
Processes." 
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EQI-MATH YEAR THREE: HOLISTIC RESPONSE CRITERIA 
Preliminary Response 



• No attempt 

- Words do not reflect the problem; 

- diagi'ams and drawings unclear or misrepresent the problem; and/or 

- does not indicate which information is appropriate to finding a 
solution. 

Begins, but fails to move toward a solution. 



Partial Response 

• Begins the problem in a manner that could lead to a solution but fails to 
complete it; 

omits a significant part of the problem; 

makes a major computational error; 

uses an inappropriate strategy for solving the problem. 

• Provides a solution, but explanation unclear; 

argument or explanation incomplete; 
diagram is inappropriate or unclear. 

(A correct answer with no evidence to support the solution must be 
further supported by an interview or else there is no basis on which to 
judge the process.) 



Complete Response 

• Gives a complete response that includes a full solution with reasonably clear 
explanations; 

communicates sound supporting arguments; 
- shows understanding of the mathematical ideas and processes. 



Elegant Response 

• Gives a complete response with an "elegant" explanation; 
unusual insight displayed in solution; 
communicates strong supporting arguments. 
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EQI-MATH YEAR THREE: SAMPLE ANECDOTAL RECORD FORM 



AJ^ECDOTAL RECORD Student No. gig 

t%m t«.c Prodlcm #l ? ' 

^ (Does not disDtay disDosraon 



Obsarvattons: 

Appears to be reading her paper. 
Draws a rapbit, erases it 
Three rabbte drawn. 
Bonding over looking under her chair. 
Looks around at people in her area. 
Tapping pencQ on taDie. 
Looks around. 

Draws more rabbits, a cfifferent view. 
Writes words. 



Interpretation: 

Not dear on wtiat she needs to do. 

Sees oi/iers drawing rai3bits and follows. 

Once tney are drawn she is not dear on where to go next 



Recommendxtlons: 

Speno more time heiping her to understarvj what the prt^blem means, wtiat is itm question 
asking. 

Needs more teacher Erected model wofidng through. 

ANECDOTAL RECORD StudMit No. 518 

btmitz ProbiMn#2 



+ - 0 



Observations: 

Writing numbers, drdes some numbers - 1 2 7i 4 5 6 7 8 9 10. 
Gets art eraser. 
Erases a9 numbers. 
Draws picture of girl 

Counts girts * 5 « adds more. 



Interpretation: 

She didnl understand the problem. 

She fen it should be numbers but then went back to draw a pjcture because she ts famt&ar 
with that 



Recommendatona: 

Model a simfiar problem for her. 

Taik her through the words to be sure that it is being understood (ESL student) > greater? 
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Eai*M«n ANECDOTAL RECQHO Student No. 

m i9Bi« Problem #4 



Observations: 



Intefpretation: 



Recommendstions: 
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PROBLEM SOLVING PRACTICES AND BELIEFS 

Responses at Beginning and End of Pilot 
October 1991 and January 1992 

8 Respondents 



How DO YOU ADDRESS PROBLEM SOLVING IN YOUR MATHEMATICS PROGRAM? 
Pllpt Teacher #7 



Before Pilot 



After Pilot 



Not very well! Try to give problems each week, 
related to material being studied; different from 
regular text work. We work through the solving 
once students have attempted them - ideas for 
strategies from students. 
1 try to model thinking processes. "This is what 
should/might trigger this approach/ 
References made to Polya's approach - on wall 
in classroom. 



Not much change her9 - no time due to 
incorporation of new M30 curricuium (attention 
paid elsewhere). Polya mociei on class wail - 
attention directed to model as appropriate 
throughout units. Teacher model - as go 
through homework, lecture. Student sharing of 
ideas - good/not so good approaches 
highlighted (from homework, problem solving, 
questions). 



Pilot Teacher #2 



Before Pilot 



After Pilot 



Students are made aware that the 
mathematical principles and concepts taught 
are mere tools to help people share problems. 
We set aside one perkxi a week for probiem 
solving. These problems are not from the text 
but chosen from other sources that have 
problems of a more stimulating nature. We 
make sure that throughout the year problems 
requiring different strategies are used. 



Problem solving is done in conjuncthn with 
each topic so that students can see how the 
topic can be applied. Sometimes I start anew 
unit with a problem that requires the new topic 
to be taught, I find the students see the need 
for what is being taught as well as generates 
Ideas on the new toph. Our tests have one- 
third of the questions in problem form. This 
shows the importance we attach to problem 
solving. 



Pilot Teacher 



Before Pilot 



After Pilot 



Problems are presented at different times, in 
different modes. A concept can be presented 
and devetoped with problems reinforcing and 
extending the concept. Some problems have 
been given as an investigation to begin 
teaching concepts. Others are given at 
random, often unrelated to concept being 
taught. 



Students have an opportunity to discuss what 
problem solving is, possible strategies are 
outlined, and other contributing factors are 
shared (number of solutions, ways to get 
solution). Problems are presented to students 
with support and encouragement to work in 
cooperating groups. Cooperation and group 
dynamics is discussed. Problems are 
debriefed with discussion including all tophs. 
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Pilot Teacher #4 



Before Pilot 


After Piiot 


Problem solving Is integrated Into all aspects of 
math as well as most other subjects. 


Problem solving is integrated into many parts of 
our day ' not just math. "Math'* problems are 
initially presented with children "acting out" the 
problem. After repeated ''acting out, " it is then 
"labelled" with numbers, eventually objects 
{blocks, beans, etc) replace the children 


PUot Teacher #5 




Before Pilot 


After Pilot 


1 do not have a regular "scheduled" 
mathematics problem-solving time. Many 
opportunities arise naturally that lend 
themselves to working through a problem. 
These problems are not always "mathematical 
problems** but are worked through in a similar 
way. The advantage of this is that the children 
are naturally engaged because It comes from 
their working situations. 


/ do not have a defined "problem-solving" time \ 
except during the pilot There are many 
occasions in math where a lesson which is 
traditionally taught can be turned into a 
problem -solving activity. To practise basic 
facts to 20 they would search for patterns and 
share with the large group, I have students 
work on standard and non-standard type 
problems and discuss "how" each group solved 
the problem. 


Pilot Teacher #6 




Before Pilot 


After Pilot 


Problem solving is introduced as a unit in 
September (first week). At this time, a method 
of solving problems is Introduced. The problem 
solving is then integraled into the other strands 
of the curriculum throughout the year. 


Problem solving is approached as a group 
project. Students work in groups to discuss 
and prepare answers to problems. Problem 
solving is included in the units taught 
throughout the year. 


Pilot Teacher U7 




Before Pilot 


After Pilot 


i teach a strategy then give examples. 1 
discuss the Houghton Mifflin daily problems. 1 
use the problem-solving cards (tags). 


/ teach a strategy but next time 1 will give the 
problem first, let them discover the strategy, 
then find solutions. Done weekly, then similar 
problems daily. 



EQI - Art & Mathematics 



1 ' 4 

C - 102 " Calgary Board of Education 
Calgary Catholic Board of Education 



PUct Teacher #a 



Before Pilot 



After Pilot 



Students are given about 2-3 problems each 
week. Initially we solved problems together as 
a class, and then I introduced a marking 
scheme that I planned to use for problem 
solving, tf students gave only the correct 
answer, then they would receive 1 mark. Using 
simpler cases and/or making a table and 
spotting patterns -2 marks; and 2 marks for 
finding a genera! rule and/or explaining why 
their rule worked. 



Students wofi( on probfems based on the 
current toph as well as 2 to 3 problems per 
w^ek, which (Day include logic problems, or 
problems involving the observations of 
pattern::. Students are required to solve the 
prob!emt, and provide an explanation about 
their soluti'>n to the problem. 



SOME PEOPLE BELIEVE THAT ALL STUDENTS CAN BE TAUGHT TO BE GOOD 
PROBL€M-SOLVERS. PLEASE COMMENT. 



P//of Teacher #t 



Before Pilot 

Good problem solvers for tlieir ability 
level, tf a method/model is available (a 
framework of reference) I believe all students 
would recognize/demonstrate some increase 
in their ability(ties) to problem solve. (They 
woukJ benefit if properly taught "how" - just like 
teachers!!) 



After Pilot 

/ agree. I believe any student can "pick up" new 
thoughts/techniques through practise and 
through teamwork which can aid in the problem- 
solving process. These new thoughts 
(processes) snowball with each experience in 
problem solving. Students recall what's been 
tried (or done) in the past and try to apply it to 
the new (even if not correct approach - it at 
least gives them a starting point). 



Pilot Teacher #2 



Before Pilot 

Problem solving is not isolated to math. All 
students have solved problems in their daily 
lives and how successful they are indicates 
how happy they are. I try to show them by 
using more personal problems how Polya's 
model works and that they can be successful in 
the math area just as they are in other areas. 



After Pilot 

Yes - all students can become good problem 

solvers. It requires: 

• confidence in one's ability 

not to be afraid of trying something new 
the basic skills he or she needs are 
understood. 

Problem solving is how they will use math later 

on. I found with a lot of patience and 

understanding, one can make a student a good 

problem solver. 



^ 1 ^ 
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PHot Teacher 



DOTore r not 


ATier rllOt 


True. Students need to be given experience in 
solving a variety of prcblems with development 
and modelling of strategies as part of the 
process. Once strategies are unders;ood, 
practice and experience are needed. 
Consideration needs to be given to motivatbn, 
sustaining interest, acceptance of challenge 
but not making process a competition. 


Students need guidance and support accom- 
panied by an opportunity to solve problems. 
Through activities, students learn possible 
strategies, work using the process and begin to 
question their knowlei and what to do. 
Teachers can support, assist and give 
opportunities to students to become good 
problem solvers. 


Pilot Teacher #4 




Before Pilot 


After Pilot 


i believe that most students can be taught to 
be good problem solvers. 


/ believe the students can be taught the 
strategies. However, I'm not sure they can be 
taught to be creative and/or confident. 


Pilot Teacher #5 




Before Pilot 


After Pilot 


1 see problem solving linked to thinking, not all 
people are effective thinkers, i do feel with 
active involvement, through time, people can 
change their view of themselves as problem 
solvers. It is a way of thinking. 


/ would now say that some students can be 
taught to be better problem solvers. 


Pilot Teacher #5 




Before Pilot 


After Pilot 


I agree, if all students are given a method to 
approach problem solving they can become 
good solvers - not brilliant ones! 


/ agree. All students can be taught to be good 
problem solvers. The key is building their 
confidence through exposing them to varied 
problem types and strategies, and allowing 
them the time to devehp this confidence. 


Pilot Teacher 07 




Before Pilot 


After Pilot 


They can learn some strategies. The slower 
ones need the most... 


We can teach the steps, but the student must 
be self-motivated and needs to internalize the 
strategy. The trying of a strategy needs more 
structure. 



1 1 f; 
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PUot Teacher #a 



netore riioi 


Aiier r^llOI 


i believe that if students are exposed to 
various strategies, and a wide variety of 
problems, they will begin to think of proWems 
as puzzles and will be able to appreciate the 
chailange, and experiencG the thrill and 
satisfaction when a solution is finally found. 


/ believe that all students can be taught to be 
good problem solvers. This can be done by 
having students solving problenns in groups, 
sharing ideas and strategies, and feeling 
comfortable that there is not one way to solve 
any problem. Students will realize that although 
their approach may be different, they could also 
find a solution to the problem. 


HOW DO YOU FACILITATE THE DEVELOPMENT OF THE PROBLEM-SOLVING 
PROCESS IN YOUR CLASSES? 


PUot Teacher #1 




Before Pilot 


After Pilot 


Not much has been done here. 


Sharing student processes with the entire 
class. Giving suggestions as to ways of 
approaching problems (in general). 


Pilot Teacher #2 




Before Pilot 


After Pilot 


We solve problems In groups - this leads to 
critical discussbn of (a) what is the problem; 
(b) a variety of possible solutions can be 
suggested; (c) which solution works or does 
not work and why; (d) what process Is required 
to put each plan into action; (e) how reasonable 
will the answer be. 


Any problems we have (not only math) are 
solved using the problem-solving method - 
example, like preparing for a test, or door- 
decorating for certain activities. After a while 
you find they tend to adopt this method for a lot 
of personal things the students do and when it 
comes to math the method is well-entrenched 
and they find it better 


Pilot Teacher #3 




Before Pilot 


After Pilot 


Sharing strategies. Working through process 
with students to develop risk-taking, open 
sessions for discussion (build talk for 
understanding) and create sense of sharing 
rather than working in Isolation. 


Students need to express their knowledge, 
become comfortable in discussing and writing 
about ideas, learn to discuss what and when 
they don't know, have models to share their 
strategies and realize they are working on 
problem solving as a process. Opportunity 
needs to be given to practise problem solving 
and sharing successes, misinterpretation and 
all the real feelings and happenings as 
problems are worked on. 
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Pilot T0ach0r ^4 



Before Pilot 


After Pilot 


Divergent type questions that are open-ended 
and allow for many possible responses are 
used (e.g. what would happen If,,.? How many 
ways can you...? Can you think of different 
ways? 


By using the process in as many areas of our 
daily classroom life as possible. 


Pilot TQachor #5 




Before Pilot 


After Pilot 


Problem so!vir>g is daily life in a year two 

r'laQQronm Prrthtom ^olv/inn in math Ho\/ftlnn<i 

as we work through problems together and 
recognize that there are many ways to get an 
answer. 1 am conscious of "step-in problem 
soiving" and strategies so i involve the 
^turiont^i in a wido ranns of nYDcripncp^s and 
through the year to introduce these. 


The discussion after the students have worked 
nift a nrohlffm /«; imcinrtunt Tiiev Sf^e how 
others did it. Tfiey rethini^ how they did theirs. 
Students then work on a similar prot)iem and 
can change the way they did it first Materials 
and conversations are encouraged. 


Pilot Toach9r #5 




Before Pilot 


After Pilot 


Modelling a successful method (Polya's). 
Reward the use of a successful method. Allow 
group (triad) problem solving. 


Group work facilitates discussion of strategies 
and methods to solve problems. 


Pilot Teacher §7 




udiore Pilot 


ATISr rllOi 


Students work alone» in pairs, in threes. 


The acceptance of all ideas by me. I try to do 
problems too, 1 share my discoveries too. I tell 
them t confer with others. I dont use an answer 
key. 


Pilot Teachor #a 




DeTore riioi 


ATier rllOl 


I try to select a variety of problems, which 1 
believe would be of interest to my students. 
Some may require experiments, or working with 
concrete materials to see patterns and 
eventually solve the problems. Hints are 
sometimes given to the whole class, or 1 may 
work with students individually, asking guided 
questions. 


Problems are given regularly. Students are 
given the opoortunity to work in groups and 
share idea'j and strategies with each other. 
Students are encouraged to read their solutions 
to the class, and solved problems are displayed 
in the classroom. 



1 J o 
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HOW DO YOU ASSIST YOUR STUDENTS TO GAIN INSIGHT INTO THEIR 
PROBLEM-SOLVING STRATEGIES? 



Pilot TeachQr #7 



Before Pilot 


After Pilot 


Using ideas of strategies to try via other 
students is the only insight used so far. 


/ gave a "debriefing" session (10 minutes) in the 
class following the problem-solving hour to 
share what 1 was seeing with all the students 
^ineir «if/fJiUiicnv9/ fjiu^ now i invu lu z»ufvtf intr 
problem, (What was good, not so good, why 
some things were better.) Also posted good 
solutions at back of c/ass. 


Pilot Teacher #2 




Before Pilot 


After Pilot 


After working in groups - each group reports on 
his solution - we then discuss this as a class 
and comnrient upon the solution. 


They wotk in groups - allows for different 
strategies to come forward. Groups present 

solutions lU Ci3SS • dyillil oflUW UlliViVfll 

strategies other groups used. 


Pilot Teacher #3 




Before Pilot 


After Pilot 


Students begin to understand strategies and 
their need in lessons specific to strategy 
development. Through practice and 
riicpuQQinnQ Qtufiont^ Qh^irp thftir *itr3tenifi^ 

with others and increase ways of developing 
more strategies. 


Illustrate, model and discuss strategies, what 
processes are involved and pose questions to 
nudge in many different directions. 


Pilot Teacher #4 




Before Pilot 


After Pilot 


Through questioning - asking for explanations. 


Through discussion and idea exchanges. 


Pilot Teacher ^5 




Before Pilot 


After Pilot 


1 have them share their method of solving and 
listen to others' ways. Explaining how one has 
"arrived" at the answer gives many places to 
engage them In thinking about their thinking. 


/ ask questhns too about what they are doing, 
1 try to make links to other problems they have 
worked on. 
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Pilot Teacher 06 



Before Pilot 


After Pilot 


Ask them to explain how they reached an 
answer, or to explain why their answer is 
reasonable. 


Providing ttiem with various problems that 
utilize differ9r}t strategies. Always asking them 
if a problem can be done another way. 


Pilot Teacher #7 




Before Pilot 


After Pilot 


They explain an answer to the class. 

Tell me what you know." 
1 have them make up problems for the class. 


/ always ask (not tell) "What dkiyou try?" "Can 
you try....?" 


Pflnt Taach^r M 

rllwi 1 Crawffvf wft 




Before Pilot 


After Pilot 


Sometimes 1 would list the different methods 
used to solve a problem, or 1 would get a 
student to explain the strategy, he/she used, 
then ask other students to explain theirs if it 
was different from the previous one read. 


/ assist my students to gain insight into their 
problem-solving strategies by having different 
strategies described by students, by displaying 
different ways in which a proioiem may be 
solved and by encouraging students to find 
other ways to solve problems. 


HOW DO YOU MODEL PROBLEM SOLVING FOR YOUR STUDENTS? 


Pilot Teacher #7 




Before Pilot 


After Pilot 


1 •^alk out" the problems; e.g. with exponent 
questbns "What do 1 see? What do 1 think? 
How do 1 progress?" 1 use them to think/be 
aware of their thinking processes and verbalize 
and write (in journals). 


Through verbal communication and 
demonstrations. This is what 1 tried... This is 
what 1 was thinking... This is how 1 kept track of 
my thoughts and plans... etc. 


Pilot Teacher ^2 




Before Pilot 


After Pitot 


1 take a problem that has nothing to do with 
math and I go through all the stages in Poly's 
model with them in solving this problem. 
Example: Inviting a limited number of people to 
your birthday party. 


Doing problems with them - asking questions 
that bring out the model to be used. Use this 
method for any problems we have as a class 
(making a team for sports events). 
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Piiot Teacher #3 



D8iore rllOl 




Rv workir ^hrounh nroblAiTi*? with students 
Prosentlng problems that do not have 
solutions. Brainstorming ideas by sharing and 
accepting suggestions. Often times problems 
are sent home and parents invited to 
participate. 


Discuss ov&rything and anything. Work on 
probiems with students. Facilitate activities. 
Support and guide students as they work. 
Pose many varied questions. 


Pilot Teacher M 




Before Pilot 


After Pilot 


Verbalize a problem 1 have - walk/lalk my way 
through to possible solutions. 


By using problems that arise in my life as a 
teacher in the classroom. 


Pilot Teacher #5 




Before Pilot 


After Pilot 


1 often think out loud while working through a 
problem. They see me struggle with "how to" 
and self-correct a redirect focus; also that this 
takes time and hQlp from others is appreciated. 


/ always encourage students to think of another 
way to solve everyday problems. When i ask, in 
math, can we solve this another way, they see 
it as part of the lesson. 1 try to help them see 
problem solving is a life skill and is flexible. 


Pilot Teacher #6 




Before Pilot 


After Pilot 


Presenting problems to the class - solving as a 
group - using guided questions. Always asking, 
"Is there another way?" 


Classroom demonstrations by myself and other 
students. 


Pilot Teacher #7 




Before Pilot 


After Pilot 


1 like to show them patterns 1 have learned. 
"Look what 1 found out." 


I do a problem solution with them. 


Pilot Teacher M8 




Before Pilot 


After Pilot 


Whenever students ask questions about any 
problem, 1 would usually draw a picture, or set 
up a table, then ask questions based on the 
diagram or the table, to try to get students to 
see relationships, and spot patterns. 


/ sometimes show students how to solve a 
problem using my strategy of drawing a 
diagram, or using a simpler case bAsed on the 
problem, and hoklng for patterns. 1 also read a 
problem several times until 1 understand what is 
being asked, and 1 list informatbn given. 1 do 
not hesitate to answer questions based on the 
approach 1 am using, and to explain why 1 used 
that method. 1 also acknowledge methods that 
are different from the one 1 applied to the 
problem. 
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• HOW DO YOU EVALUATE STUDENT PROBLEM SOLVING? 



Pilot Teschor #1 



Before Pilot 


After Pilot 


1 try to use a scale of 5 (holistic marking). Very 
little has been done by me in this area. 


With difficutty? Folbwod the 0-1 -2 approach of 
preliminary, partial, complete response (and 
elegant) but still had difficulties... 


Pilot Teacher i2 




Before Pilot 


After Pilot 


1 look for (1) does the student understand the 
problem; (2) did he make a plan and follow rt 
through; and (3) how original is his solution. 


See last sheet. 


Pilot Teacher #3 




Before Pilot 


After Pilot 


Student evaluation is based on observation 
and interaction in class -contributions to 
discussion, participation, interest and work 
within the process, ability to acquire the 
language and use it. 


By obsen/ation, contributions during 
discussion, knowledge and feedback in oral 
and written form, strategies as shared during 
debriefing sessions. 


Pilot Teacher #4 




Before Pilot 


After Pilot 


V^hen they're working with concrete material 1 
observe and record. Later when they're 

WUi^lll^ ax lilc; oyiMLAJllO aiiU dUoLlct^L oici^c?^ i 

rely on student products 


Through obsen/ation and some student 
product. 


Pilot Teacher #5 




Before Pilot 


After Pilot 


1 currently do not have a specific way to 
evaluate problem-solving growth and change. 1 
did watch and note good strategies, who gets 
frustrated, who sticks with it, who needs more 
clarification than average. 


The way they reached their answer is what 1 
look at. Also, how they talk about getting the 
solution. 1 also like to see what they do when 
they get stumped. 



i r r 
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Before Pilot After Pilot 



With a focused holistic scoring point scale (0 to 
4). 


/ use a four-point system: 

0 - blank; 

1 - correct answer, no work; 

2' correct strategy but hcomplete, 

inappropriate strategy - incorrect answer; 

3 - appropriate strategy but wrong answer; 

4 - correct strategy and clear explanation 


F//of Teachor #7 

Before Pilot After Pilot 


Worth 4 points: 1 plan. 1 trying, 2 correct 
answer. Make a problem = 4 points. 10% of 
math mark. 


a) 0' 1 points for each part understand 
strategy, answer, explain 

b) Problem of the week - a candy 

c) Tests on similar patterns 


PUot Teacher ^8 

Before Pilot After Pilot 


Correct answer only - 1 mark 
2 marks if strategy was shown 
(diagrannlable) and recognizing pattern or 
relationships, and 2 marks for explaining 
why the njle works/drawing conclusions 
from the diagrams/patterns. 


1 evaluate student problem solving 
holistically, by giving 1 mark for a 
Preliminary response; 2.5 marks for a 
Partial response; 4 marks for a Complete 
response; and 5 marks for an Elegant 
response. 


IS THERE ANYTHING ELSE YOU WOULD LIKE TO ADD? 
Pilot Teacher #t 

Before Pilot After Pilot 


I have done little problem solving in my 
classroom (with "word" problems). 1 wish to 
l^enefitfrom materials/Ideas from others since 1 
doni have the time/direction to establish 
materials myseif. Tm open to trying something 
new! 1 look fonnrard to the wealth of 
knowledge 1 will gain from this project! 


If students gave a complete problem-solving 
approach and answered the question, even 
though the answer was incorrect, 1 still gave 
them "Complete Response" rating but 1 
constantly struggled with that. 
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PUot Taachmr ^2 



8efor> Pilot After Pilot 

/ found the journal writing on the students 
picked very hard - as I was not privileged to 
what they were actually thinking of -but merely 
based my opinions on obsen/ations which at 
times are misleading. 



PUot Teacher #3 



Before Pilot 

Problem solving needs to be considered not a 
separate area of development but an integral 
process of program. Many teachers group 
problem solving with strands for development 
rather than as a process. 



After Pilot 

Teachers and students often need to break the 
bonds of searching only for a solution. Key 
areas of development the teachers and 
students need to collaborate on are group 
dynamics, cooperative learning, metacognition, 
and the ability to realize how you think to use 
your abilities most effectively (even 
recognizing what you dontknow), problem- 
solving strategies and process. 
Word problems are difficult to use at the Grade 
2 level as often it is not the problem-solving 
dispositions you end up evaluating, but rather 
the student's reading ability. 



Pilot Teacher #4 



Before Pilot 

The very nature of learning mathematics is 
problem solving. We learn math so we can 
function in life and that functioning is more than 
figuring word problems. If math is taught from a 
problem-solving perspective, then we are 
teaching math for life and students learn it 
because it is meaningful and they understand it 
well as its value. 



After Pll- ^t 

This project has made me do more formal 
problem-solving lessons, ft also helped me 
track students differently, ft gave me a more 
definite way to categorize what f saw. 



Pilot Teacher #5 



Before Pilot 

I thought the anecdotal records were valuable 
but rd rather be involved by mediating the 
student's view of the problem. 



After Pilot 

Time is the most important factor in problem 
solving. Each child must be given time by you, 
encouraging and guiding them to carry out the 
problem-solving process. They also need time 
to develop a method. 
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Pilot Teacher #5 



Before Pilot After Pilot 



1 teach a strategy lesson once a week and give 
sample questions. The other days i supply 
cards, etc., for the groups. Before report cards 
1 give sample tests. 


/ try to emphasize 'Whfch strategy do you plan 
to try? Which strategy did you use?" to try to 
get them to develop a plan. 


Pilot Teacher #7 

Before Pilot After Pilot 


1 am continually looking for new strategies and 
Interesting problems which would help to make 
my students better problem solvers. 


1 would also like to include interviews into the 
evaluation process for problem solving. 
Interviews will help the students to clarify the 
strategies used, and enable the students to 
give a more complete solution to the problem. 



• CAN YOU !DENTIFY ANY WAYS YOU HAVE CHANGED YOUR TEACHING OR 

ASSESSMENT PRACTICES AS A RESULT OF PARTICIPATING IN THE EQI-MATH 
PILOT? PLEASE EXPLAIN BELOW. 



Pilot Teacher #r 

After Pilot 

I see the benefit of private interviews to discuss student progress on an individual basis but still 
doni see how to fit it in. 

I continue to model thinking processes (and planning, acting out plan, etc., stages). 

I have not changed teacher/assessment practises yet, but EQI has opened my eyes to 

possibilities that I'd tike to incorporate - now I must figure out how. 

The assessment scales are good - quick and easy in general - a checklist would be most 

appropriate. 

I found the 6 one-hour sessions were too precious In a semester to really "give up** and still would 
prefer a more "inherent" approach in the Math 10 program (re: problem solving) - though how? I 
must still try to figure out. 

Thank you for the opportunity to be a part of this project. I enjoyed learning! Hopefully, I can now 
put things into practice (slowly but surely). 



Pilot Teacher #2 

After Pilot 

There are some new wrinkles in my approach now: 

1 . Letting the students read the problem aloud to themselves, as well as to someone else, 
seems to help them to understand the question better. Also, the interaction of the two leads 
to better comprehension and more ideas (strategies). 

2. Marking problems using the holistic approach was better received by the student. They find 
the results more encouraging and are apt to try where before, they were afrakl to. Makes 
them better risk-takers. 

3. Has given me a better insight on how they approach problems and what areas of difficulty we 
must work on. 
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PUot Teacher 03 

After Pilot 

The project brought a problem-solving focus to my class by supplying me with problems. I felt a 
need to explore, Investigate and practise, more Immediately, much of what had been discussed In 
class prior to th^ project. I was also looking for more Innovation to work with students. The area 
of metacognltlon arose and became very useful for both me and the students. For students who 
have not had opportunities to discuss and build understanding, talk assists and improves their 
ability to formulate ideas and express them In written form. The project Increased my awareness 
and knowledge of group process, group dynamics, problem-solving process, and the use of 
crHrcal thinking within a realistic context. The students made progress, would identify and 
discuss what they were about. Sounds like progress for me and the kids!!! 



PUot Teacher #4 

After Pilot 

I believe I do a better assessment now by using the mathematical dispositions as a checklist 
guide. _ 



PUot Teacher #5 

After Pilot 

I would like to include a problem-solving time where students record their '^thinking" as In pilot. 
This was helpful in assessment. 1 can see advantages In working on all types of problems. I am 
sure I have missed many gooo problems and teaching opportunities. The students may benefit 
from a more consistent "time" to practise problems in a formal way. h gives a wide range of what 
has come up in a natural way. Til have to try it and see. 1 wouW continue to use the Holistic 
Response Criteria to sort responses. It allows you to see strengths and weaknesses. 



Pilot Teacher 46 

After Pilot 

I have realized that discussion in a necessary part of the learning process in problem solving. 
Students must vocalize their thoughts in order to evaluate their appropriateness. It's somewhat 
like working at the concrete level, students must hear It to understand it. Another important thing 
that IVe learned is that children need time to digest the problem-solving model. Some children 
might take hours and, if you try to speed up the process or cut H back, you will inhibit their ability 
to learn. 

I wouki begin problem solving with lots of problems, talking and very little writing. 

Be more conscious of the process rather than the end product. Give them plenty of problems at 

the beginning of the year and not mark them (no risk involved). 

Pilot Teacher #7 

After Pilot 

! will spend more time on k>oking for strategy problems. 1 will vary the strategies more in shorter 
units than repeat the units. 
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Pilot Teacher 08 



After Pilot 

As a result of participating in the EQI-Math pilot, I have changed some of my teaching and 

assessment techniques. I use a period per week for problem solving in groups, as well as 

assigning 2 to 3 problems each week for students to work on individually. 

I have had short informal interviews for students whose problem-solving skills are at the 

preliminary and partial response stages. (Hoping to organize a more formal set of interviews next 

semester.) 

I display the students' solved problems each week. I now have materials such as drce. markers, 
squared paper, geometry sets, tape, geometrk; shapes, etc., available in the classroom, rather 
than retrieving them from the cupboards on demand. 

I now mark holistically because it gives a clearer prcture of the strategies students are using, and 
I also have a better idea about which students would benefit from interviews. 
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RESULTS OF A SURVEY OF TEACHER PARTICIPANTS IN 

THE EQI-MATH PROJECT 
MAY 1992 



I PARTICIPANTS 

Twenty-four classroom teachers were involved in the development and 
pilot testing of materials for the EQI-Math project. Sixteen of these 
teachers helped to develop the materials during the 1989-90 and 1990-91 
school years. Five of the 16 were involved for only one year as a result of 
transfers to different schools and other assignments. Eight teachers pilot- 
tested the materials during the 1991-92 school year. 

A survey was sent to all 24 teacher participants. Seventy-nine percent (19) 
were returned and processed. 

II PURPOSE 

The purposes of the survey were to determine (a) teacher practice in 
teaching problem solving, and (b) the impact of the EQI-Math project on 
teaching practice as reported by the participating teachers themselves. 
Another major aspect of the survey was to determine areas of the project 
that were particularly successful, and areas that should receive additional 
attention in the future. 

IIL RESULTS OF THE SURVEY 

A. Teacher practice in problem solving 

Teachers were asked to indicate, on a scale of: none - little - some - 
much -undecided, the degree to which they made use of 15 teaching 
techniques which contribute to the development of student problem- 
solving skills. Teachers reported that all but one were using "some" or 
"much" in their teaching practice. 

The exception was: Having students take problems home, thereby 
inviting parent participation, which received a rating of ^^little - 
some.'' The others, in order of usage from ''much'' to "some/'were: 

Techniques receiving "much" usage: 

- Ask students to explain how they reached their answers. 

- Have students working in pairs or groups. 

- Illustrate, model and discuss strategies. 
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- Talk out problems indicating, for example, what I see, what I think, 
how I would proceed. 

- Encourage students to acknowledge methods that are different 
from the ones they applied to a panicular problem, 

- Present problems that have more than one solution. 

- Provide students with a variety of problems which will require 
utilization of different strategies, 

- UtiUze brainstorming techniques. 

- Have students present solutions to the class. 

- Have students demonstrate to others or to the class. (This received 
"much" emphasis in Grades 5 and 8, "some emphasis in Grade 2, and 
"least" emphasis in Grade 10,) 

Techniques receiving ^'some'^ usage: 

- Have debriefing sessions in which students can talk about what 
was successful or not successful in their problem-solving 
approaches. 

- Apply a problem-solving model to a real-world problem that has 
nothing to do with Mathematics. (Received "much" emphasis in 
Grades 5 and 8, "some" emphasis in Grade 2, and "little" emphasis in 
Grade 10.) 

- Work through problems with individual students. 

- Have students make up their own problems, (This received "some" 
emphasis at the elementary level and between "little" to "some" at 
the secondary level.) 

Comments 

With the exception of the qualifications noted in three of the items 
above, there was considerable consistency in the utiUzation of the 15 
techniques among teachers from all grade levels. 

It was not possible to collect "before" and "after" types of data on 
teacher utilization of these problem-solving techniques as a result of 
participation in the EQI project. However, further study might be useful 
in determining whether the use of the EQI-Math assessment materials 
encouraged the increased use of particular teaching strategies. 
Additional study would be useful in examining the connection 
between teaching and assessment strategies and student success in 
problem solving. 

B. Impact of EQI-Math on participating teachers 

- Teachers were asked to respond to the following statements on a 
seven-point scale: strongly agree - agree - agree somewhat - 
undecided -disagree somewhat - disagree - strongly disagree: 
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Participation in the EQI project has been a valuable! useful learning 
experience for me. 

Response: 

• All respondents reported in the "agree" categories: 68% "strongly 
agree," 26% "agree," 5% "agree somewhat." 



My involvement in EQI has made me a better teacher of mathematics. 
Response: 

• 95% responded in the "agree" categories: 58% "strongly agree," 
26% "agree," 11% "agree somewhat." 5% (one respondent) was 
"undecided." 



/ believe that participation in the EQI project was the type of 
experience that would be of value to teachers of subject areas other 
than mathematics. 

Response: 

• 95% responded in the "agree" categories: 63% "strongly agree," 
32% "agree." 5% (one responded) "disagreed." 



The problem-solving materials developed in the EQI project are: 

a) useful to me 

Response: 

• 80% responded in the "agree" categories: 37% "strongly agree," 
42% "agree," 5% "agree somewhat." 5% were undecided and 11% 
did not respond to the item. 



b) welhorganized and presented 
Response: 

• 84% responded in the "agree" categories: 42% "strongly agree," 
37% "agree," 5% "agree somewhat." 5% were undecided and 11% 
did not respond. 
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c) easily understood 



Response: 

• 85% responded in the "agree'' categories: 32% "strongly agree," 
53% "agree." 5% were undecided and 1 1% did not respond. 



d) appropriate for the grade level(s) I teach 
Response: 

• 84% responded in the "agree" categories: 37% "strongly agree," 
42% "agree," 5% "agree somewhat." 5% were undecided and 11% 
did not respond. 



Comments: 

Undecided and non-respondents to the above question might be those 
who did not see the finahzed materials used during the pilot. Pilot 
materials were not sent to any of the teacher researchers. 

• Teachers were asked: 

To what degree has your participation in the EQI project increased 
your skills and confidence in the following areas of mathematics, and 
a list of 7 activities followed. 

Response: 

- Teachers in all grades reported that participation in the project 
increased their skills and confidence most greatly ("much") in the 
area of "assessment of student growth in problem solving." 

- All of the other areas received a rating of "some" increase in 
confidence and skills: 

• discussing progress in mathematics with students 

• skills in teaching problem solving 

• knowledge of problem-solving techniques 

• reporting student progress in mathematics 

• using new teaching strategies in mathematics 

• discussing children's progress with parents 
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MATH PROBLEMS USED IN PILOTING 



EQI2 

Problem #1 

Mary Jane saw 8 rabbits. 

How many ears did those rabbits have in all? 

Show and explain all your wort 

Circle one of the following words you would use to describe this problem: 

Boring Challenging Confusing Fun 

Circle one of the following words that describes how you feel after trying to 
solve this problem: 

Discouraged Satisfied Frustrated Happy 



EQI5 

Problem #5 

David and Jason are visiting Calgary. 

One day while exploring the city they realize they are lost. 

David said to Jason, "When we were at the Calgary Tower we were 8 blocks 
west of the hotel." Jason remembered that after they left the tower they went 
south 4 blocks, then east for 3 blocks, north for 2 more blocks, then east for 5 
blocks. 

How far away from the hotel are the boys when th ey realize they are lost? 
Show and explain all your work. 

Circle one of the following words you would use to describe this problem: 

Boring Challenging Confusing Fun 

Circle one of the following words that describes how you feel after trying to 
solve this problem: 

Discouraged Satisfied Frustrated Happy 
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EQI 10 

Problem #4 
A. 

Take any four consecutive numbers; multiply them together and add 1 . 
Take the square root of the number. 
Is your final answer a whole number? 

Show and explain all your work, 
B. 

Take the middle pair of numbers and multiply them together. 
Compare with your first answer. 

C. 

Repeat parts A and B three more times and write down a possible rule. 



Cii'cle one of the following words you would use to describe this problem: 

Boring Challenging Confusing Fun 

Circle one of the following words that describes how you feel after trying to 
solve this problem: 

Discouraged Satisfied Frustrated Happy 
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1.0 BACKGROUND TO THE EVALUATION OF PROJECT MANAGEMENT 



The Calgary Educational Quality Indicators (EQI) Project was a three-year action research 
project (1989 - 1992) funded by Alberta Education to develop indicators of quality student 
performance in school art composition and mathematics problem solving at the Calgary 
Board of Education and the Calgary Catholic Board of Education, 

The project purposes were defined as follows: 

L To identify indicators of quality work which can be applied in the assessment 
of student achievement in areas of Art and Mathematics 

2. To identify conditions that appear to enhance or limit the achievement of 
quality 

3. To develop strategies for collecting quality student work which include 
recorded interviews with teachers and students 

4. To examine quality work with emphasis on alternative assessment strategies 
to paper and pencil testing 

The projea was part of a larger, province-wide initiative sponsored by Alberta Education 
to build sets or clusters of indicators to use in a broader approach to assessment. A number 
of other jurisdictions were involved as well, looking at a variety of assessment topics. 

The Calgary project focused on assessing performance in the areas of art and mathematics. 
These subject areas were chosen because of preliminary work which had already been done 
at the Calgary Catholic School Board in the area of art assessment and because of 
mathematics expertise at the Calgary Board of Education Further, it was reasoned that the 
two areas could allow a unique exploration of the assessment of product and process. The 
an project could examine the product of the learning, as much of art education had stressed 
process in the past, and the math project could examine the process of student thinking, as 
most math education had emphasized the product, or answer, in the past 

The project design incorporated three phases: development, field testing and pilot testing 
of quality indicators. Committees of teachers and subject spedalists reviewed student work, 
identified charaaeristics of quality work, considered ahemative approaches for recording 
assessment information and proposed activities and materials for use in classrooms. 
Classroom teachers tried out a variety of materials and approaches with their students, 
materials were revised and then were piloted with other teachers. 

The project had a fiill-time designated director who was located at the Calgary Board of 
Education. A steering committee, comprised of school superintendents and representatives 
from Alberta Education provided coordination and leadership, A community advisory 
committee, comprised of university professors, leaders in the fields of an and mathematics, 
parent representatives, representatives of the Alberta Teachers* Association, and members 
of the business conmiunity provided a sounding board to test project ideas. Finally, two 
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subject area advisory committees, with specialists from both boards, provided "hands on' 
decision making for the action research in each area. 

The art component of the project culminated in the development of the Educational Quality 
Indicators Handbook which included colour reproductions of quality samples of students' 
an from Grades 2 and 5, and from junior and senior high along with teacher expectations 
for each lesson and student comments assessing their own work* Three quality indicators 
emerged from the project as observable characteristics to be found in students* work, 
including re/afion5/z/p5, handling and meaning. Student and teacher journals were used in the 
pilot test to assist in reflective practice on the part of both teachers and students. 

Project conclusions for the art component included the following: 

1. Quality assessment praaice supports improved art making- 

Z Attending to written and/or spoken student responses, reflections and 
thoughts illuminates the art work, 

3. Quality art assessment is a shared experience between students and teachers. 

4. Effective art assessment depends on common vocabulary shared by teachers 
and students. 

5. The collection of quality art exemplars emphasizes diversity as an attribute of 
a quality art program. 

6. Expert professional judgement benefits from collegia] work. 

7. Assessment takes time. 

8- EQI Handbook exemplars are a way to share expertise, to develop 
connoisseurship. 

Tlie math component explored the concept of holistic scoring and developed some response 
criteria including preliminary response, partial response, complete response and elegant 
response. A series of students* mathematical dispositions were identified, including 
motivated, creative, confident, strategic approach and strategic process. Conditions for the 
math learning enviroimient were also considered. 

Projea conclusions for the mathematics component included the following: 

1. Teachers report positive changes to their teaching and assessment practices 
as a result of participating in EQI-Math. 

2. Teachers want to meet together to discuss teaching and assessment. The 
experience of working closely with colleagues is very positively received. 
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3. Useful materials and practices developed in the project included student 
dispositions, holistic scoring and the exploration of diverse student responses 
and quality solutions to mathematical problems. 

4. The use of anecdotal records, student attitude reports and generalizations 
about classroom conditions was not seen as practical, di^e to time and energy 
constraints. 

5. Students seemed to be more positive about problem solving, took more risks, 
and demonstrated more confidence as a result of the EQI project. 

6. Systematic methods for observing students closely helped teachers with 
teaching decisions and with assessment and evaluation tasks. 

7. Students respond to the invitation to express themselves more fully and to 
think about their thinking. 

Finally, some commonalities emerged about the making of art and the solving of 
mathematical problems in terms of the complex thinking processes required for each and 
also in terms of the assessment challenges they present teachers. Common findings mcluded 
the following: 

1. Quality assessment practice in art and mathematics supports teacher decision 
making about teaching and assessment 

2. Attending to written and spoken student responses, reflections and thoughts 
illuminates the art work or the problem solution. 

3. Quality assessment is a shared experience between teachers and students 
which is encouraged by shared exemplars and a shared vocabulary. 

4. Qualitative assessment takes time for both teachers and students. 

5. Exemplars are a way to share expertise, develop connoisseurship, establish 
common expectations for student outcomes and celebrate diversity of student 
response. 

The evaluation of project management at the local level was mandated to be an 
independent, third-party evaluation and as such was contraaed out to an independent 
program cvaluator. 

A series of research questions was developed in collaboration with the Project Dirertor. 
They vvere as follows: 

LO UTILITY 

1.1 How useful are the EQI elements to assess student achievement? 
12 Can they be used for reporting to parents? 
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13 Can EQI materials be applied to other subject areas? If so, what 
areas? 

1.4 How would you model or depict key elements? 

2.0 PROJECT MANAGEMENT 

2.1 Were the necessary management systems in place to support successful 
projea implementation? 

22 What impact did the steering committee have on project 
implementation? 

23 What impact did the advisory committee have on project 
implementation? 

2,4 Did the partnership of the two school systems affect either project 
implementation or outcomes? K so, how? 

3.0 PROJECT IMPACTS/RECOMMENDATIONS 

3,1 What recommendations would you make to improve project utility? 

32 What recommendations would you make to improve project 
management? 

33 What happened as a result of the EQI project that you did not expect? 

3.4 What impacts will the EQI project have on education in Alberta? 

A series of eleven face-to-facc interviews were held with representatives of the two school 
jurisdirtions and Alberta Education. A breakdown is as follows: 



Calgaiy Board of Education Senior Administration 1 

Project Director 1 

Math Education 2 

Art Education 2 

Calgary Catholic School Board . Senior Administration 1 

Math Education 1 

Art Education 2 



Alberta Education Senior Manager 1 

The data were compiled and analyzed for content after Krippcndorff (1980). The remainder 
of this report is a summaiy of responses for each of the research questions outlined above 
followed by some conclusions. 
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2.0 RESEARCH FINDINGS 



Each of the research questions will be explored in turn. The comments were grouped and 
analyzed according respondents' areas of expertise (art, mathematics or supervision), and 
according to employer (Calgary Board of Education, Calgary Catholic School Board or 
Alberta Education) to search for similarities or differences. Where appropriate, these 
similarities or differences will be identified. 



1.0 UTILITY 



1.1 How useful are the EQI elements to assess student achievement? 

Generally, respondents found the art EQI elements to be very useful. The art specialists 
agreed that both the visuals in the handbook and the concepts of meaning, relationships and 
handling would be valuable to both the specialist and the generalist The fact that the 
student was present for the assessment process and able to talk in a shared language about 
the artwork was seen to be very positive as insight could thus be gained into how to make 
that artwork stronger. The distinction between assessment and evaluation was seen as a 
useful one. However, it was pointed out that some orientation would be required for both 
teachers and students to interaa successfully with the handbook. 

The math specialists were divided in their views about how useful the math EQI elements 
would be to assess smdent achievement On the positive side, it was seen that the elements 
provided some useful language to describe what teachers should be looking for in problem 
solutions and what dispositions students should exhibit As one specialist commented: 

The holistic approach makes us look at the solutions to problems and not just at the 
answers. 

Another aspect of the project which was considered useful was the opportunity it provided 
for teacher collaboration. 

However, the progress made in the math component was seen as •'a first small step*", 
consistent with current thinking in math research, but the project did not provide any 
conclusive links between student dispositions and exemplary work. As one individual 
commented: 

Tliis is not the definitive answer, it is some information. 

Two of the four supervisors interviewed were positive in their responses, the others were 
more qualified. The elements were seen to be useful because they reflected emerging 
educational thought about reflective teaching, self-assessment, authentic assessment and 
performance assessment As one conmiented: 

Viere are new ways of seeing, new words to use and new ways to gather informatioru 
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However, it was also felt that the materials would have to gain respect on their own merit. 
Teachers were inundated with materials and would have to see the handbook as instantly 
useful and app icable if they were to explore it farther. It was the belief of one supervisor 
that math was a higher priority than art for elementary teachers and therefore the art 
handbook was less4ikely to be adopted by them, whereas art teachers in high school were 
specialists and might be more likely to use the elements. Another supervisor felt that the 
art handbook would be used but that as the outcomes in math were less tangible, they might 
be less likely to be utilized as a result The fear was expressed that the issue of 
institutionalizing projea concepts had yet to be addressed and that the lack of involvement 
on the part of senior administration in both boards was a deterrent to implementation. 

Overall, with the exception of the art supervisors from both boards who were positive in 
their responses, respondents from the Calgary Board of Education viewed the project 
elements as being more useful than those from the Calgary Catholic School Board. 

12 Can they be used for reporting to parents? 

The art specialists were positive in their views that exemplars of students' art in the 
handbook would be very useful in talking to parents as it was seen as a good place to begin 
a dialogue about their children's work* In fact, pilot teachers indicated that this was the 
case. One teacher had reported a conversation with a parent as follows: 

. . . /le realized for the first time how powerful art is in a child's learning. 

It was felt that the EQI handbook would be used as a model for anecdotal reporting in the 
future. 

The math specialists were divided in their views about using EQI elements for reporting to 
parents. One felt that a lot more understanding had to be developed about how children 
learn Math before performance assessment would be an acceptable reporting format or 
teachers' credibility would be at risk. In addition, this individual felt that parents were not 
yet ready to deal with the qualitative nature of this approach. The other, however, quoted 
a pilot teacher who had said: 

Wlien I did my report cards, I knew my lads better this time. 

This individual felt that looking at dispositions and extended solutions was helpful in 
reporting to parents. 

Generally the supervisors were positive in their view that EQI elements could be used for 
reporting purposes. They saw both parents and the business community as being open to 
new forms of assessment. However, probably due to their administrative perspective, they 
had several suggestions about what would have to happen first These included: 

1. The need for parent partnership in their child's assessment 

2. The need for guidelines for performance assessment available 
to the public 
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3. The need for public and parent education 



They saw a number of advantages to using alternative assessment tools, including parent 
ownership in the child's assessment, a shared understanding of quality in the community and 
a means of acloiowledging risk taking and alternative solutions, not just correction and 
control. One supervisor did feel, however, that reponing would have to take a different 
format from those developed in the project before it would be ready for the public but 
another hoped that the elements would be used for reporting and commented: 

Far more parents want this kind of evaluation than do those who want a letter 
grade. When I show them the richness of information from the fads' ''talk" they 
are positive. With thoughtful discussion with parent groups, it has great promise. 

An analysis by employer group did not reveal any differences in opinions overall between 
the two boards. 

13 Can EQI materials be applied to other subject areas? If so, which 
areas? 

This question elicited a number of comments on areas of transfer. Every respondent felt 
that there were lessons to be learned from the EQI project and that either parts or all could 
be transposed to other subject areas. 

Two of the math supervisors felt that both the quality indicators and the project process 
could be applied to any subject. As one commented: 

Teachers of all subject areas need to identify problems for students^ get responses, and 
sit down as educators to look, ready analyze and discuss student responses and to set 
benchmarks about realistic expectations for certain ages ofchildreru The process to find 
out about what teachers think about responses can be done in any area. 

In addition, the interaction which had occurred between the art and math projects was seen 
as an example of transferability: 

Tliey have words to describe the art and they can now be applied to math. Math could 
take the reflection component from art. It would be the next step to have kids reflect on 
their solution and give them the words to talk about solutions (self-assessmerU). You 
could take it anywhere. It is valuable to assess the process. 

The art specialists saw a particularly close link between art, writing and social studies. A 
similarity was seen between the way art was edited and critiqued in the EQI project and the 
way students' writing can be assessed and edited so that students take ownership of the 
process. The use of positive exemplars was seen as another transferrable instructional 
method as was the use of journals to provide the oppormnity to capture reflection. As one 
individual commented: 

Children are looking for meaning. ... 77ie insight into students' reflection on their work 
is very ricfu 
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Social studios was seen to have a strong link to historical and visual materials so that the 
EQI concepts of meaning, relationships and handling were seen to be applicable to culture 
and environment as well 



The supervisors tended to agree that the EQI concepts were transferrable. In addition, the 
view was advanced that they related to inter-subject areas as well such as English and social 
studies where the use of portfolio assessment was already in place: 

It fits to have them look at dispositions, as the math dispositions are really learning 
dispositions. Art lias des^eloped generic criteria. The invitation to let students assess 
themseWes and to develop criteria about what is successful to internalize quality 
staruiardSy show samples^ use vocabulary to look at work in progress [all fit]. 

It was pointed out that there already were writing standards developed and that different 
subjects, such as industrial arts, already used samples of finished products as benchmarks. 
In addition, teachers used holistic indicators to assess project work (e.g., a puppet piay for 
a book report assignment). A close link was seen between math and science in terms of 
problem solving skills. 

To summarize, project concepts were seen to be transferrable to the following subjects (with 
frequency of comment presented in parentheses): 

All subjects (2) 

Language arts (2) 

Science (2) 

Social studies (1) 

Inter-subjects (1) 

However, two respondents had reservations. One felt that the project had developed 
generic impressions but that each subject area would have to develop specific descriptors. 
However, this individual did believe that the dispositions and response criteria developed 
in the math component had some applicability to other areas. 

The other respondent believed that a prototype had not yet been pulled out of the project 
that could be generalizable. While the project had been an excellent experience for those 
involved in it, the EQI findings could not in their current form find widespread applicability. 

An analysis by employer group did not reveal any differences in opinions overall between 
the two boards. 

L4 How would you model or depict key elements? 

This question was difficult for respondents to answer. No one came up with a definitive 
model for the project. However a number of components were identified which could be 
massaged into a model which would depict the following: 
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Student-Teacher Interaction Students and teachers involved in a meaningful, 

equal, reflective, critical activity using shared 
language in order to problem solve and add 
furtiier understanding 

While no one had a complete model, some interesting components were identified which 
could be extracted for further analysis. These included the following: 

♦ Inter-teacher dialogue, analysis and collaboration regarding student responses 

♦ Samples of exemplary work to develop a shared understanding of quality 

♦ Exploration of communications about problem solving 

♦ Use of journals to explore student reflection to access the richness of student 
thought (mirrors of the mind) 

♦ The paucity of information provided by traditional paper and pencil testing 

♦ The relationships between the amount of space provided to capture student 
responses on paper and the quantity of reflective comments elicited 

Staff members from the Calgary Board of Education seemed to have more suggestions 
regarding a potential model than other respondents did. 

2.0 PROJECT MANAGEMENT 

2.1 Were the necessary management systems in place to support successful 
project implementation? 

A number of positive comments were elicited regarding the management and management 
systems for the implementation of the EQI project These included the foUowing (with 
frequency of response in parentheses): 

n^^alitv nf Prniert Direction (4) 

The Project Director was cited for her excellent job, sometimes in trying 
circumstances, in terms of background reading and preparation, keeping 
everyone on task, writing and editing, and external communications. The 
project was seen to have run smoothly largely due to her direction. 

Adequate Budget (2 pro. 1 con) 

The budget was seen to be adequate to run the project by two respondents 
but the third described it as "bare bones". However, the project was never 
hindered from lack of money and was even able to produce the art Handbook 
with costly reproductions. 



Ariftqnatft Time (2^ 



The three years were seen to be adequate for project development. There 
were comments, however, that one more year would have been helpful to 
disseBiinate and institutionalize the concepts. 

Facilities an d Secretarial Support (1 each't 

The location of the project in the Calgaiy Board of Education's head office 
was seen as helpful in that adequate space and seaetarial support were 
provided. However, it was the view of the projert director that she might 
better have been located on site wi± one of the project components. 

I^^tworking (I) 

The housing of the project director in the Calgary Board of Education's 
Department of Curriculum allowed her to be hooked into a variety of 
curriculum networks at both the local and provincial level which facilitated 
the flow of information between the project and its larger environment 

On the other hand, a numb^^ of problems related to project implementation were also cited. 
These included the foUov. jg: 

Turnover in Project Representat ion in the Math Area (4^ 

While the art representation and leadership remained constant and strong 
throughout the project, the math component suffered from the turnover of 
several members, including the curriculum leader from the Calgary Board of 
Education and the specialists from the Calgary Catholic School Board. It was 
felt that the turnover slowed the process, and led to a lack of clarity regarding 
project goals and boundaries. It was felt that the math team never "clicked" 
and had less commiunent to the project which had a negative impact on 
project outcomes. 



I^ck of Involvement of Board Sup flrintendents (3) 

Respondents commented that the superintendents of both boards did not take 
ownership for the project. Despite their involvement in proposal development 
and the acquisition of funding for the project initially, they were only 
minimally involved at the Steering Committee level (see below) and did not 
make the project a high priority. The project director was seen to be on her 
own with an orphan project to run. This was contrasted with other provincial 
projects where district superintendents demonstrated ownership and 
involvement It was feared that this lack of engagement and buy-in would 
have an impart in terms of the institutionalization of projert concepts. 
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T^ck of Involvement of Project Initia tors in Project Implementation (2^ 



The initial proposal was written by central office personnel from the two 
school boards, including a member of the Research and Testing Department 
of the Calgary Board of Education (now the Assessment Team), with 
consultation from one art specialist. The proposal writers never worked on 
the project and project advisors had trouble clarifying project goals. Time was 
lost interpreting what they had in mind and was particularly problematic for 
the math component 



Project Dire ctor Position (1> 

The fact that the project director held a term-specific assignment which 
terminates at the end of the EQI projea rather than holding a line position 
in either board was also seen to be a deterrent to potential future 
institutionalization of projea concepts. 

An analysis by employer group did not reveal any differences in opmions overall between 
the two boards. 

22 What impact did the steering committee have on project 
implementation? 

A project steering committee was comprised of the two board superintendents, the subject 
area leaders, representatives from Alberta Education, and the project director. The purpose 
of this committee was to monitor project developments and receive and approve project 
reports. Two meetings were held during the initial phase of the three-year project, but then 
the function of this committee was incorporated into existing advisory conunittees. The 
Alberta Education representative helped connect the projea with the provincial EQI 
director for decision-making. 

Identification of membership on the steering committee was limited. The various steering 
and advisory committees were confused in most respondents* minds in terms of both 
membership and funaion. The projea direaor's contraa stated that she was to report to 
the steering committee regularly and to receive feedback and advice from the committee. 
Interim projea reports were sent to the provincial co-ordinator as part of the projea 
contraa and feedback was received. 

One issue which was discussed at the steering committee level was that of qualitative versus 
quantitative analysis of study outcomes. The provincial EQI mitiative advocated a balanced 
approach to measuring outcomes. The Calgary projea focused more on qualitative methods 
and, therefore, some quantative data which were colleaed during the projea may not have 
been included in the final report 



An analysis by employer group did not reveal any differences in opinion overall between the 
two boards, 

23 What impact did the advisory committees have on project 
implementation? 

There were three advisory committees. One had community representation, and the other 
two were working committees in the subject areas. As a result, responses varied depending 
on the perspective of the respondent 

The art advisory committee was seen as having a major impact on the project* The project 
made decisions about the project, guided implementation, debriefed after project activities, 
and interpreted results. It was from this level that the action research emanated and the 
process was seen to work very well: 

Tlie teachers found it to be a rewarding professional experience to sit with experts and 
talk about teaching art, methodologies and philosopfiies. They referred to it as a 
collegia! model and recommended itasaPD model They then used the same model 
with the child The answers were not there at the beginning. Teachers became inventors 
and researchers and became part of the process* 

In addition, it was suggested at the Art advisory committee that journal writing be 
incorporated into the project. This turned out to be one of the positive results of the 
project 

The math advisory committee was also seen to have a major impact on project direction, 
although this was mitigated to a degree by lack of continuity. The committee was seen as 
a positive force in the project and committee members were involved with teacher in- 
services and classroom implementation. It met frequently to react to proposed math 
problems and student responses. As one member of the team commented: 

We kept the project going in a current direction in tune with the literature in math 
education. I don't think teachers would have done that on tlteir own. We were able to 
stand back and take a look at new directions. 

The third community advisory committee was seen by the project director as a good 
sounding board to try out project ideas on a wider audience. 

It appears, overall, that the two subject area advisory committees were the heart of the 
project, in conjunction with the classroom teachers and students who implemented their 
ideas. 

An analysis by employer group did not reveal any differences in opinions overall between 
the two boards. 



2.4 Did the partnership of the two school systems affect either project 
implementation or outcomes? If so, how? 

Respondents were positive in their views about the impact the partnership between the two 
boards had on the "project. Positive outcomes included the following (with frequency of 
responses in parentheses): 

It provided an opportunity for new linkages and net working (6) 

It was seen as particularly valuable that teachers from the two boards, who 
did not know each other prior to the project and who had few opportunities 
in the past to work together, could come together for professional dialogue. 

It was a good example of a partnership (6) 

The project gave the two boards a reason to conraiunicate in a positive way 
in depth about a professional topic. This was seen as a unique experience. 

It was a positive professional developme nt experience (2^ 

Project involvement resulted in an increase in the overall base of expertise 
among project teachers and advisors* 

It allowed two different systems to work together (2) 

The two school systems were seen to have very different organizational 
structures and management styles. The aoss-fertilization of ideas was seen 
as a positive outcome. 

There were few negative outcomes identified. It was mentioned that as the projert director 
was housed at the Calgary Board of Education, it necessarily followed that the bulk of 
project costs were supported by this board. However, it was also noted that staff from both 
boards cdntributed equally to the project equal numbers of students were involved, and an 
equal amount of ownership was demonstrated* 

An analysis by employer group did not reveal any differences in opinions overall between 
the two boards. Both were equally positive about the experience. 
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3.0 PROJECT IMPACTS/RECOMMENDATIONS 

3.1 What recommendations would you make to improve project utility? 

The art specialists Had few suggestions for improvement of the project and generally felt 
very pleased with the outcomes. It was suggested, however, that the an handbook binder 
format could be improved upon and the suggestion was made that an accompanying video 
of teacher-student interactions would be helpful as well. 

The math specialists reflected on the fact that their component had started at a less 
developed level of conceptualization than the art component and as a result had farther to 
go: 

They were looking at the product. We had to look at the process and had never done 
much in that area. The product in math is a lot simpler. In art there is no right answer 
so they had explored process already. We got teachers focusing on the solution as a 
documentation of the thought process. To improve project utility would be to have kids 
talk or write about their solutions. Documentation of kids' reflections on their solutions 
would be the next step for us. 

In fact this individual suggested that another project was required in math to allow 
continued pursuit of these preliminary findings. 

Another respondent asked what would have happened if the students had generated the 
math questions rather than the teachers, and further, whether or not the teacher-generated 
questions were the right ones for each grade level. This individual feh that a lot more 
exploration was required in this area. 

Another considered the frustration experienced in participating in this developmental project 
in contrast to the typical mathematics educator's experience which focused on the bottom 
line. The amount of time invested in the project vidth little demonstrable output was a 
concern. 

The supervisors had a number of suggestions to improve project utility from their 
perspective. These are summarized below. 

Ensure implementation at the Cal garv Board nf Education (4) 

Due to both differing structures within the two boards and to recent financial 
decisions made at the Calgary Board of Education, it was felt that the 
dissemination of projea concepts was threatened at the CBE because of lack 
of personnel to spearhead the process. As the Assessment Team had not 
been involved in the project, it was not anticipated that this area would 
continue the development of the project in the future. The subject area 
specialists were being reassigned due to budget constraints and so would not 
be available to ensure that project concepts were utilized. The project 
director's tenure would be up at the end of the 1991 - 1992 school year and 
so no further direct leadership was envisioned. It is anticipated that 
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curriculum personnel at the Calgary Board of Education will utilize the 
findings of the project as they work in their inter-disciplinary team 
assignments in schools. 

On the other hand, the Calgary Catholic School Board was retaining their 
subject area specialist structure, and four of the sixteen supervisors had 
express instructions to integrate projea concepts and practices into their work. 

It was hoped that some son of partnership arrangement might continue so 
that CBE teachers could access in-service activities in this area at the Catholic 
board. 

Ensure that support he provided on project concepts (4^ 

It was felt that very carefully designed teacher support would be required to 
help teachers work with EQI concepts. It would be important for discussion 
to be encouraged among groups of teachers, that mentoring programs be set 
up to allow for on-going dialogues among pilot and other teachers and that 
principals be provided with learning oppoitunities so that they could support 
teachers in this process. The outcomes of this project were seen as a possible 
basis for a revision of student assessment but the process would have to be 
nurtured. It was also suggested that parents would require background 
information and that a parents' primer be developed for use at Parents' 
Advisory Councils. 

Overall, employees of the Calgary Board of Education had more recommendations to 
improve project utility than did employees of the Calgary Catholic School Board. Not 
surprisingly, the representative from Alberta Education strongly wished to see project 
concepts built into practice in the district. 

32 What rwommendations would you make to improve project 
management? 

Responses to this question tended to reflect comments made earlier in the interview process. 
Two majbr recommendations were advanced regarding how project management could have 
been improved, as follows: 

involvement of s enior level sta keholders (3) 

Lack of on-going involvement of senior management at both boards and 
Alberta Education was seen as a drawback to project implementation- As a 
general thnist toward performance assessment is perceived in educational 
circles, it was felt that this project should have received more attention than 
it did. 
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Tnvnlvgment of profect initiators (2^ 



While several of the same individuals were involved as in the above 
recommendation, the lack of continuity of vision between the project 
designers -and the project implementors was also seen as a drawback. 

A number of other recommendations were advanced by individual respondents about what 
might have occurred to improve project management These can be grouped as follows: 



Recommendations abou t project management 

Integrate concepts sooner into other subject areas 
Increase number of classrooms involved 
Increase interaaion time for teachers 



Recommendations about the m ath component 
More consistency of vision 

Look at a few quality solutions rather than a class set 

Review math problems for ambiguity, appropriateness of level of complexity, 

sequence from simple to complex 

Have students develop math problems themselves 

Support teachers in developing anecdotal reportu^ skills 

Support teachers in developing consistency when marking 

Support teachers in making judgements which are based on agreed-upon 

criteria 

There were no recommendations advanced to improve the management of the art 
component as the specialists were pleased with the process. 

An analysis by employer group did not reveal any significant differences of opinion although 
views were influenced by internal issues. The representative from Alberta Education wished 
there could have been more of a focus on the provincial goal of building a set of mdi<ators 
for widespread use. There was a tendency during the project to lose sight of the provmaal 
context. • 



3 J What happened as a result of the EQI project that you did not expect? 

The surprises respondents experienced tended to cluster around either process issues or 
project outcomes. 

Unanticipated aspects of the process included the following: 

The push for quantitative data, the later abandonment of it, and the 
willingness with which math teachers accepted qualitative assessments (3) 
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Lack of teacher consistency in marking (1) 

The ease of the art component coming together and the difficulty of the math 
component in doing likewise (1) 

Unanticipated aspects related to project outcomes included the following: 

The journals and the richness of the information they provided (3) 

New appreciation for students as artists or collaborators (2) 

New ways for students to critique their work and talk to their peers (1) 

Increased diversity of student artwork (1) 

The impact of using language instead of letter grades in math (1) 

The solidarity of findings between art and math (i.e., more generalizable 
characteristics) (1) 

The lack of conelation between the dispositions and achievement (1) 

Other administrative surprises have already been referred to in 3.1 and 32. 

An analysis by employer group did not reveal any differences in opinions overall between 
the two boards. 

3.4 What impact will the EQI project have on education in Alberta? 

Respondents* views varied about project impacts. One voice indicated that the project 
would not have a provincial impact: 

No. A lot of money was put into it and not to come out with something clear is 
bothersome. . . . We need to highliglu things we learned about student tfiinking and 
teticher thinking. 

The larg6srnumber of comments were positive in nature. Typical examples are as follows: 

Hopefully the project had influenced the assessment model in Alberta. It has helped to 
break down barriers, to bring people out of isolation, and to raise their understamUng of 
the interaction of visaed and language. 

Wlxen the student can articulate why their work is good, tften the next time they can work 
on it and advance iL There was a consensus about vision that made the project 
successful, the fact tlxat we were able to define quality and its factors. 
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A lot of teachers said they really learned, thai they got to know their students better as 
individual learners. 

It was an enjoyable project. We haue learned a lot. There was excellent assistance from 
the teams. They read a lot on assessment and caused us to network with people from 
other school systems. I wish we had another year. 

It fiad a marvellous empowerment effect on teachers. 

It was a wonderful opportunity for school system collaboration. 

Other comments related to whether or not the products which were produced were 
worthwhile. Some were convinced of theu- value: 

It has already had a substantive impact in the talk in school jurisdictions. . . .Some new 
language emerged from this project, new demonstrated proof that teachers are reliable 
knowers and valuable information on student-teacher relationships. 

Tills study gives people a tangible example of the theory. We have a product and have 
practised applying it in concrete situations. It ffves people some starting groimd to start 
their own performance assessmenL 

We are giving them tools and exemplars to use. The handbook will make teachers more 
reflective Samples of art have been judged excellent by those who know. In mathy kids 
thought in their heads and wrote down the answer. Now we are asking them to write 
about the process. 

But others had a more qualified view: 

It depends. If they produce the book, if they produce a video, if they incorporate it into 
a curriculum package, it wiU have an impact. It had an impact on all the teachers who 
were involved. Teachers who used it could substitute, add their own exemplars and 
collect their own students work Students liked to see what students before and after 
them (Le., younger and older) could do. 

It depends on how user friendly the final products are. The teachers have so many 
choices tluit it has to be exceptional material It is a quality project but it may be 
overwhelming for teachers, particularly at the elementary level where they have 7 - 8 
subjeas to prepare for every day. Tlie art EQI handbook is pretty overwhelmuig It is 
impressive but intimidating. I fiave a subtle caution at the back of my mind that 
sometimes teachers waru something simpler. 
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Others saw the project as just one step in a long progression towards performance 
assessment: 

Once the report is published it will be one more piece of evidence to support the thrust 
towards cha^jgingthe delivery of mathematics in our schools We must change it. The 
project will support movement about how we assess students and I would like it to open 
the door to dialogue about math and its place in education as more than the third R 
It is as important to be literate in math as it is to be literate in language. 

I would hope it would lend support to those in assessment for valuing the process and 
the solution as opposed to the answer. Lots of people are pressing to look at this. It 
becomes a piece of good information that we consider. There is lots of neat stuff 
around. Thb adds to the value m math learning. We won't he doing workshops on EQI 
but we can't help but mention it if we are talking about assessment. 

We are moving in the direction of performance assessment 

However, the EQI project was also seen as a positive influence on the acceptance of 
qualitative research in Alberta: 

We even had an influence on Alberta Education staff. They wanted numbers and 
research support* What we were doing was much more subjective and powerful . . . 
Tlie way we did it was more of an action research approach. The qualitative research 
is legitimate and people thought about it. I would Iwpe it would lend support to those 
in assessment for valuing the process and the solution as opposed to the answer. 

It is not fast enough for me but it is having a major unpad on valuing naturalistic 
iruptiry. 
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3.0 CONCLUSIONS 



Respondents at the local level felt that the EQI project had resulted in several useful 
concepts in both an and mathematics which reflect emerging thought in the area of 
performance assessment The art concepts and visuals were seen to be very useful in their 
current format while the math concepts were seen more as a jumping off point for further 
development. Again, due to the greater advancement which had been possible in the art 
component, the handbook was seen as a useful tool for discussions with parents, while the 
math concepts were felt to need further development before they could be discussed with 
parents. However, it was feU that the dispositions identified and the idea of exploring how 
students arrived at solutions was already helping teachers understand their smdents better. 
Respondents agreed that EQI concepts and the EQI project process could be transferred 
to other subject areas. Assessment m math and science was seen to be linked as it was in 
art, language arts and social studies. The potential for assessing cross-subject assignments 
was seen as another application. None of the respondents had developed a model to depict 
EQI concepts. Perhaps this is not surprising as these concepts were neither linear nor 
sequential hi nature but were rather interactive and evolutionary. 

A number of useful perspectives were advanced with regard to project management The 
du-eaor was commended for the excellent management of the project The resources, 
support and time allotted to the project were judged to be appropriate, y^eas which were 
problematic included the turnover of mathematics personnel on the project the lack of 
involvement of senior level administrators at both boards, and the lack of involvement of 
most of the projea initiators in its implementation. The conunittee structure and 
membership was not clearly understood by a number of project participants. The steering 
committee met infrequently, although some policy direction was provided at that level. The 
community advisory committee acted as a sounding board for the project director. It was 
at the subject area advisory committees that the action occurred-most projea decisions 
were made here and implementation of project concepts in schools was durected from these 
groups. In addition, these advisory committees provided the vehicle for the networking 
which was viewed so positively by all involved. The partnership between the two boards was 
seen as very positive in that it also encouraged new networks and linkages. The project was 
described as an excellent professional development activity for all involved. 

Recommendations to improve project utility in the art component related to upgrading the 
binder fcJrmat of the handbook and developing videos of classroom use to accompany the 
handbook. In the math component it was generally felt that the concepts explored and 
developed during this projert should provide the basis for additional research in the future. 
As one respondent commented, "We need another year." General recommendations related 
to ensuring that project findings would be implemented at the Calgary Board of Education 
despite the reorganization planned in the curriculum area next year, and to providing 
appropriate in-service opportunities for teachers on EQI concepts. Respondents felt that 
project management would have been facilitated both at the start-up phase and for the 
institutionalization of project concepts at its conclusion if senior administrators, project 
initiators and the CBE assessment team had been involved- Unanticipated outcomes 
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identified by respondents related mainly to the unexpected richness of findings and their 
utility and transferability, as well as to a new appreciation for students as collaborators in 
the assessment process. Overall, it was hoped that the EQI project would have a positive 
impact on performance assessment practices in Alberta in general and in the Calgary area 
in particular. It was also evident that this would take time to accomplish. 
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